Ultrasound image processing to render three-dimensional images from two-dimensional images

ABSTRACT

Methods for processing two-dimensional ultrasound images from an intracardiac ultrasound imaging catheter provide improved image quality and enable generating three-dimensional composite images of the heart. Two-dimensional ultrasound images are obtained and stored in conjunction with correlating information, such as time or an electrocardiogram. Images related to particular conditions or configurations of the heart can be processed in combination to reduce image noise and increase resolution. Images may be processed to recognize structure edges, and the location of structure edges used to generate cartoon rendered images of the structure. Structure locations may be averaged over several images to remove noise, distortions and blurring from movement.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No.14/095,628, filed 3 Dec. 2013 (the '628 application); which is acontinuation of U.S. application Ser. No. 13/290,696, filed 7 Nov. 2011(the '696 application), now U.S. Pat. No. 8,622,915; which is acontinuation of U.S. application Ser. No. 11/772,161, filed 30 Jun. 2007(the '161 application), now U.S. Pat. No. 8,057,394. The '628application, the '696 application, and the '161 application are eachhereby incorporated by reference as though fully set forth herein.

BACKGROUND

The present invention relates to medical diagnostic systems and methods,and more particularly to methods for rendering three-dimensional imagesfrom two-dimensional images generated by an ultrasound imaging cathetersystem.

Recent advancements in miniaturization of ultrasound technology hasenabled the commercialization of catheters including phased arrayultrasound imaging transducers small enough to be positioned within apatient's body via intravenous cannulation. By imaging vessels andorgans, including the heart, from the inside, such miniature ultrasoundtransducers have enabled physicians to obtain diagnostic imagesavailable by no other means.

Due largely to their small size, ultrasound imaging transducers used toimage from the inside of the heart render two-dimensional slice image“frames”. These image frames are generally bounded by the maximumimaging depth within an image scan angle. Typically, the scan angle isapproximately 90 degrees, while the image depth depends upon theultrasound frequency and the power.

While two-dimensional image frames provide very valuable diagnosticinformation, they require the clinician to mentally integrate many imageframes taken at different rotational orientations in order to imaginehow the heart appears in three-dimensions. In many medicalcircumstances, the clinician would benefit from being able to view theheart in three-dimensions.

While it has been suggested that two-dimensional ultrasound image framesmay simply be stitched together to assemble three-dimensional images, apractical system for accomplishing this does not exist due to thedifficulty of such image processing. Many factors, both physiologicaland technical, have prevented the combination and assembly of imageframes with the degree of accuracy, detail and reliability required forcardiac diagnostic purposes.

SUMMARY

The present invention provides effective methods for processing andcombining two-dimensional ultrasound image frames in order to generatethree-dimensional and four-dimensional (three physical dimensions plustime) composite images and cines in normal cardiac cycles, as well as indiseased states, where the motion of the cardiac muscles might not berhythmatic, and the frequency of such motion might not be within thespatial or temporal sampling capacity of the imaging set up. Theembodiment methods include, but are not limited to, recognizingstructure edges within ultrasound image frames using edge detectionalgorithms, and determining the coordinate locations of the detectedstructure edges. Nominal or average locations of heart structures arecalculated by averaging (or other statistical measure) the coordinatelocations of detected structure edges in multiple image frames obtainedfrom the same viewing perspective. Averaging the coordinate locations ofedges in multiple ultrasound images provides a single average locationfor structures that are moving during imaging. The averaged structureedges from images at a particular transducer rotational orientation arethen combined with the averaged structure edges from images at otherrotational orientations in order to generate a three-dimensionalapproximated cartoon rendering of heart structure within the imagedvolume. Such methods can be combined with selective imaging or selectingimages for processing based upon measured electrocardiogram signals togenerate three-dimensional average cartoon renderings of heart structureat various points in the heartbeat cycle. Such three-dimensional averagecartoon renderings datasets can be used to provide the clinician with aninteractive display of heart structure.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitutepart of this specification, illustrate exemplary embodiments of theinvention, and, together with the general description given above andthe detailed description given below, serve to explain features of theinvention.

FIG. 1 is a block diagram of an intracardiac ultrasound imaging system.

FIG. 2 is an illustration of an intra-cardiac ultrasound imagingcatheter positioned in the right ventricular cavity.

FIG. 3 is a diagram of an intracardiac ultrasound catheter withtransducer array and temperature sensor.

FIG. 4 is a diagram of the imaging area of an intracardiac ultrasoundtransducer array.

FIG. 5 is a diagram of an intracardiac ultrasound transducer arrayshowing its positional and orientational degrees of freedom.

FIGS. 6A and 6B illustrate ECG signals of a normal and diseased heart.

FIG. 7 is an illustration of an intra-cardiac ultrasound imagingcatheter positioned in the right ventricular cavity illustrating thephenomenon of multipath interference.

FIG. 8 is a flow diagram of an embodiment method for determining thetransducer position and orientation by processing ultrasound images.

FIG. 9 is a flow diagram of an embodiment method for time-gatingultrasound images with a patient's electrocardiogram.

FIG. 10 is a flow diagram of an embodiment method for processingultrasound images to enable construction of a three-dimensional image.

FIG. 11 is a flow diagram of an alternative embodiment method forprocessing ultrasound images to enable construction of three-dimensionalimages.

FIG. 12 is a flow diagram of an embodiment method for processingultrasound images to register imaged structure within an externalcoordinate reference frame.

FIG. 13 is a flow diagram of an alternative embodiment method forprocessing ultrasound images to enable construction of three-dimensionalimages.

FIG. 14 is a flow diagram of an embodiment method for processingultrasound images to enable a user to obtain detailed image data of aselected portion of an imaged organ.

DETAILED DESCRIPTION

Various embodiments of the present invention will be described in detailwith reference to the accompanying drawings. Wherever possible, the samereference numbers will be used throughout the drawings to refer to thesame or like parts or method steps.

As used herein, the terms “about” or “approximately” for any numericalvalues or ranges indicate suitable dimensional tolerances that allow thepart or collection of components to function for their intended purposesas described herein. Also, as used herein, the term “patient” refers toany human or animal subject and is not intended to limit the systems ormethods to human use. Further, embodiments of the invention will bedescribed for use with an intracardiac ultrasound transducer arraycatheter. However, the embodiments may be applicable to any medicalultrasound transducer and are generally useful for ultrasound imaging ofany portion of a patient's body.

Typical ultrasound imaging catheter systems, particularly intracardiacultrasound imaging catheters, generate two dimensional sliced images oftissue, referred to as image frames, within the field of view of thetransducer array. Since the ultrasound imaging catheter has a smalldiameter, such as about 6 to 10 French, it can be inserted into mostorgans of the body via catheterization through a vein or artery, orthrough small incisions such as in an arthroscopic procedure. Forexample, an intracardiac ultrasound catheter can be introduced into theheart through the vena cava to image the atria and ventricles from theinside. Such access of the imaging sensor provides image details andperspective that are available by no other imaging means.

The main elements of an embodiment of an intracardiac ultrasound imagingsystem are illustrated in FIG. 1. The illustrated system embodimentincludes an ultrasound transducer array 22 carried by or positioned on acatheter 20 coupled to an ultrasound unit 40 by a signal cable 28. Theultrasound unit 40 is connected to a display, such as a display unit 70,by a data interface 75 which may be wired or wireless. Exampleultrasound imaging system embodiments suitable for use with the presentimage processing method embodiments are disclosed in U.S. patentapplication Ser. No. 11/610,778, entitled “Integrated Beam Former AndIsolation For An Ultrasound Probe” filed Dec. 14, 2006, the entirecontents of which are hereby incorporated by reference.

A signal cable 28 delivers ultrasound signals from the ultrasound unit40 to each of the transducers in the array 22. Typically, the signalcable 28 will include at least one wire per transducer, and in anembodiment, includes a coaxial cable connected to each transducer in thearray 22. Typically, the signal cable 28 includes an electricalconnection plug (e.g., a standard connector) on its proximal end.Providing a plug connector on the proximal end of the cable 28 allowscompletion of the many electrical connections between the cableconductors and the ultrasound unit 40 by pressing the plug into acomplementary connector in the housing 100 of the ultrasound unit 40.

The transducers in the array 22 convert the electrical signals from theultrasound unit 40 into sound waves, which propagate into a portion of apatient's anatomy, such as the heart. The same transducer array 22 alsoreceives ultrasound echoes reflected from anatomic structures andtransforms the received sound into electrical signals (e.g., by means ofthe piezoelectric effect). These electrical signals are conducted viacable 28 back to the ultrasound unit 40.

A signal generator 46 generates electrical signals of ultrasonicfrequencies to be provided to the ultrasound transducer array 22. Thesignal generator 46 can be configured to produce signals of particularwave forms, frequencies and amplitudes as desired for imaging tissue.The signal generator 46 is configured to generate signals with thenecessary transducer-to-transducer phase lag to enable the transducerarray to generate a focused and steerable sound beam as is well known inthe art of imaging ultrasound phased array transducers. Alternatively,phase lag may be added by another circuit, such as a beam former circuit54.

A transmit/receive multiplexer circuit 48 can be included to direct thesignals generated by the generator 46 to isolation circuitry 44 and toseparate out echo signals returned from isolation circuitry 44 from thegenerated signals.

Isolation circuitry 44 is included to isolate unintended, potentiallyunsafe electrical currents and voltages from the transducer array 22which contacts the patient. Also, a thermal monitoring circuit 42 and acut-off circuit 43 may be included to mitigate possible risks to thepatient that can result from excessive local heating by ultrasound. Anexample of such safety methods and systems is embodied in the ViewMate®catheter ultrasound system from EP MedSystems, Inc. of West Berlin, N.J.

A filter and conditioner circuit 51 can be included in the ultrasoundunit 40 to reject spurious signals that may be induced in or throughcable 28.

An analog-to-digital converter (ADC) 52 can be included in theultrasound unit 40 to frequently sample and convert the ultrasoundsignals from analog electrical levels to discrete digital numericvalues.

A signal buffer 53 can be included to store at least a portion of theecho signals, which are returned from the transducer array 22 and whichmay be processed by other elements of the ultrasound unit 40. In anembodiment, a signal buffer 53 is included to store the echo signals asdigital data in a random-access semiconductor memory (RAM).

Beam former 54 circuits may be included to process signals sent to andreceived from the transducer array 22 to enable phased-array ultrasoundimaging. The beam former 54 may receive ultrasound signals from thesignal generator 46 and introduce phase lags for each transducer elementso that when the signals are applied to the transducer elements a narrowbeam of sound emanates from the array. Also, the beam former 54 mayreceive signals from the transducer array and process the ultrasoundecho signal data to calculate the amplitude and direction of theultrasound echoes returned to the transducer array 22 from each of manyspecific angles and distances. The beam former 54 may also determine thefrequency or Doppler frequency shift of the signal returned form each ofselected angles and distances from the transducer array 22.

In an embodiment associated with cardiac imaging, the ultrasound unit 40may also include electrical connections for receiving signals fromelectrocardiogram (ECG) electrodes and for passing such signals on to anexternal electrocardiogram or electrophysiology unit 60 which may beconnected to the ultrasound unit 40 through a communications interface62. The communications interface 62 may be any wired or wirelessinterface. In an embodiment, the ECG electrodes can be an intracardiacelectrophysiology catheter 64 which includes one or more electrodes 66near a distal end for sensing electrical activity in the heart.Electrical signals sensed by the electrodes 66 can be conveyed to theultrasound unit 40 by means of an extension of the catheter 64 or aconnecting cable 68.

In an embodiment, signals sent by the ECG or electrophysiology unit 60through the interface 62 can be recorded or used to synchronizeultrasound image data with the heartbeat of the patient. For example, asequence of images may be associated with a sequence of ECG readingsrevealing the phases of the cardiac cycle, or images may be capturedonly at a specified phase of the cardiac cycle as explained below withrespect to FIG. 9.

In some embodiments, the image display unit 70 can convert theultrasound data generated by the beam-former 54 (which may be relativeto a transducer-centered polar coordinate system) into an image relativeto another set of coordinates, such as a rectangular coordinate system.Such processing may not be necessary in the display unit 70 if theconversion was already preformed in the ultrasound unit 40. Techniquesfor converting image data from one coordinate system into another arewell-known in the field of mathematics and computer graphics.

FIG. 2 depicts a simplified cross section of a human heart 12 with anultrasonic imaging catheter 20 positioned in the right ventricle 14. Thecatheter 20 includes an ultrasound transducer array 22, which can imageat least a portion of the heart 12. For example, the image viewing angle26 afforded by the transducer array 22 may allow imaging the leftventricle 13, the septum 16, the ventricular walls 15, 17, and othercoronary structures from the right ventricle 14. Insertion of thecatheter 20 into a circulatory system vessel or other anatomical cavityvia percutaneous cannulation is well known in the medical arts.

FIG. 3 illustrates the distal end of a typical ultrasound imagingcatheter 20. The transducer array 22 is typically positioned near thedistal end of the catheter 20 behind a linear ultrasound window. Also, atemperature sensor, such as a thermistor 24, will be positioned in thecatheter 20 near the transducer 22 and electronically coupled to athermal monitoring circuit 42 to permit the system to cut off power tothe ultrasound transducers if the measured temperature exceeds a safelimit. Examples of phased array ultrasound imaging catheters used inperforming intracardiac echocardiography and methods of using suchdevices in cardiac diagnosis are disclosed in the following publishedU.S. patent applications—each of which is incorporated herein byreference in their entirety.

2004/0127798 to Dala-Krishna et al.;

2005/0228290 to Borovsky et al.; and

2005/0245822 to Dala-Krishna et al.

An example of a commercially available ultrasound catheter is theViewFlex® available from EP MedSystems, Inc. of West Berlin, N.J. Itshould be noted that the present invention is not limited to thespecific catheter assembly disclosed in the applications cited above,because the invention is applicable to various ultrasound imaginginstruments designed for intravascular and intracardiacechocardiography.

The ultrasound echo return signals are radiofrequency signals that aretypically converted to digital form and stored in memory 53. Memory usedby an ultrasound imaging system may be a semiconductor random-accessmemory (RAM) or hard disc memory of the system or coupled to a networkserver. The beam-former 54 processes the returned echo signals receivedfrom the transducer elements to determine the time of arrival from whichthe distance to the source of each echo can be determined. Techniquesfor beam-forming for both transmission of the ultrasound pulse andreception of ultrasound echoes are well known in the fields ofultrasound imaging and in phased array ultrasound, sonar and radar.

The result of processing of the data stored in buffer memory 53 by thebeam-former 54 can be a pixel-based image relative to a polar-coordinatesystem spanning the imaging angle. To generate such pixel-based imagedata, the amplitude, phase and time of arrival of reflected ultrasoundpulses at each transducer in the array 22 are processed in thebeam-former 54 to yield an RF signal reflecting the distance andintensity of echoes along the axis of the emitted beam, the angle ofwhich (with respect to the long axis of the transducer array 22) isknown. The distance and angle data may then be combined with amplitude(i.e., power of the received echo) to produce a pixel of image data(e.g., by processing the data according to an algorithm). Alternatively,the beam-former 54 may store or output the processed received ultrasoundas datasets comprising data groups of angle, distance and amplitude. Inthis and like manner, the beam-former 54 can turn the large volume ofstreaming ultrasound signals into a smaller set of data easily passedover a data link 75 for processing or display by a display unit 70.

The beam former 54 may also compare the frequency of the generatedsignals with the frequency spectrum of the returned echo signals. Thedifference in frequency relates directly to the velocity of tissue orblood toward (higher frequency) or away from (lower frequency) thetransducer array due to the Doppler Effect. The difference in frequency,i.e., the amount of Doppler frequency shift, is indicative of motion ofthe tissue (including blood) from which the ultrasound is reflected. Thefrequency shift may be determined by comparing the generated signal andreceived echo signal and detecting the difference in frequency. Theconversion of the frequency shift to velocity depends on the speed ofsound in the body, which is about 1450 to 1600 meters per second in softtissue, including blood. The conversion of frequency shift to velocityaccording to well known algorithms may be performed immediately by thebeam former 54, or later, such as by an external image processor at thetime the image is displayed. If calculated by the beam-former 54, thevelocity data (or Doppler shift) may be outputted as a fourth element ofthe dataset, so that echo sources are identified by angle, distance,amplitude and velocity (or frequency or Doppler shift).

The beam former 54 or the ultrasound unit 40 may directly produce animage in rectangular coordinates. Alternatively, the beam former 54 orthe ultrasound unit 40 may produce an image in polar coordinates andtransform the image into rectangular coordinates. Alternatively, thebeam former 54 or the ultrasound unit 40 may simply produce an image inpolar coordinates (i.e., angle and distance coordinates) and allowsubsequent image processing to perform a coordinate transformation asneeded (such as in image display unit 70).

A buffer memory 53 may make available the return signal datarepresenting the ultrasound echo waves, and the beam-former 54 mayaccess that data to calculate the amplitude of the ultrasound echo ateach of many specific angles and distances from the transducer array.

A programmed microcontroller, microprocessor, or microcomputer 41 orfunctionally equivalent discrete electronics can be included tocoordinate the activity described above within the ultrasound unit 40.In addition, the microcomputer 41 (or equivalent) may respond toconfiguration parameters and commands sent from the image display unit70 over the communication interface 75 or 76 to the ultrasound unit 40.This microcomputer 41 within the ultrasound unit 40 may be in additionto a system processor, which is a programmable computer, such as aworkstation or laptop computer, that is electronically coupled to theultrasound unit 40. In such configurations, the microcomputer 41 mayreceive configuration and control instructions from the system processorwhich can have a user interface (e.g., display with menus, pointerdevice and keyboard). In some system configurations, the activities ofthe ultrasound unit 40 may be controlled directly by the systemprocessor.

In an embodiment, the ultrasound unit 40 may be configured via softwareor discrete circuits to adaptively cut and separate each image frame ofultrasound image data. Such capability may be used to select andtransmit frames for which there is useful information (e.g., changes inposition of structures) to limit the bandwidth required for transmittingultrasound images to external displays. In a normal cardiac cycle,portions of the heart are at rest for significant fractions of thecardiac cycle, so numerous images during such intra-contraction periods,illustrated as durations 600 in FIG. 6A, will contain the same imageinformation. Alternatively, images may be selected during theintra-contraction periods 600 for processing since the relativestability simplifies image processing and combining of images. By nottransmitting images from portions of the heart beat cycles, the desiredimage information may be transmitted at substantially lower data rates.Such processing of image frames may be accomplished by a segmentationmodule (not shown).

In an embodiment, signals from an ECG sensor such as anelectrophysiology catheter 66 may be used in lieu of, or in addition to,signals from an external ECG unit 60, which may have its own ECG sensoror sensors. The ECG sensor signals can be used to record or control thetiming of the ultrasound image acquisition relative to the cardiac cycleinstead of or in conjunction with signals from an external ECG unit 60.The signals from an ECG sensor may be included within the data streamoutputted by the ultrasound unit 40.

In addition to including connectors for receiving the input/outputconnection plugs for ultrasound catheters and ECG sensors or equipment,some embodiments of the ultrasound unit 40 include connections foradditional sensors, such as intracardiac percutaneous leads,subcutaneous leads, reference leads and other electrical leads that maybe employed during a procedure.

A scan converter 82 may be used reformat polar coordinate image datainto an image relative to a rectangular coordinate system as needed.Image data from the scan conversion (and Doppler processing, if any) maybe processed by an image renderer 83, then formatted and displayed as animage on a video monitor 73. For example, the rendering circuit 83 maygenerate a gray-scale image (such as a B-mode image) in which thebrightness of each pixel is representative of the amplitude of theultrasound echo from the anatomical small volume to which the pixelcorresponds.

The image display unit 70 may perform other functions. For example, theinteractive control 80 in the image display unit 70 may transmitconfiguration parameters and control commands to the ultrasound unit 40,where the configuration parameters and commands may be supplied by theoperator by means of interactive inputs from a pointing device (mouse,trackball, finger pad, or joystick, for example) and a keypad orkeyboard 72 attached to the display unit 70. Optionally, the interactivecontrol 80 of the image display unit 70 may forward the image and/or rawdata to a network file or database server, to the Internet, to a displayscreen, or to a workstation through a communication interface 92.

In an embodiment, the image display unit 70 circuitry may be includedwithin the ultrasound unit 40 housing or chassis. This may beaccomplished by simply including the image display unit 70 circuitry asanother board or VLSI chip within the ultrasound unit 40. Alternatively,the circuitry and functionality of the components of the image displayunit 70 may be incorporated in a VLSI chip that also encompasses thebeam-former 54 and/or microcomputer 41 within the ultrasound unit 40. Insuch an embodiment, one or more of the various image processing methodembodiments describe below may be programmed into and be performed bythe image display unit 70 circuitry or a microprocessor within theultrasound unit 40. In such an embodiment, the ultrasound unit 40outputs the processed image data as a video signal (e.g., VGA, compositevideo, conventional television or high-definition video) that can becarried by a cable 75 directly to a display 73 to yield an image on thescreen without further processing. In a further embodiment, theultrasound unit 40 may output processed image data as a networkcompatible signal, such as Ethernet or WiFi, that can be directlycoupled to a network.

One or more display monitors 73 may be included as part of ultrasoundunit 40. Any of many choices, sizes, and styles of a display 73 may beconnected to the ultrasound unit 40. For example, the external displaymonitor 73 may be a cathode ray tube, a liquid crystal display, a plasmadisplay screen, “heads up” video goggles, a video projector, or anyother graphical display device that may become available. The displaymonitor 73 may be large and may be located conveniently out of the way,such as a plasma screen hung from the ceiling or on a wall. The displaymonitor 73 may be positioned for better viewing by the physician, andmay be positioned remotely, such as in another room or in a distantfacility. The display 73 may be connected to the ultrasound unit 40 by acable, an infrared link, a radio link (such as Bluetooth), or anyequivalent wireless technology.

In an embodiment, the display monitor 73 and/or the user input device 72may be embodied by a computer terminal, workstation, or personalcomputer such as a laptop computer. Such an embodiment can be configuredto display the graphical output from the image rendering circuits 83 andto pass user inputs on to the interactive control 80 of the ultrasoundunit 40. Alternatively, in an embodiment in which the display monitor 73and user input device 72 are provided by a computer system, the computersystem may operate software enabling it to perform one or more of theimage processing method embodiments describe below on the data receivedfrom the ultrasound unit 40.

As useful as intra-organ ultrasound images can be to a clinician, theimages obtainable from a catheter mounted ultrasound imaging system arenecessarily limited to two dimensional slice (i.e., cross-sectional)image frames. This limitation to two-dimensional imaging results fromdimensional limitations inherent in a catheter ultrasound imaginginstrument. On the one hand, an imaging catheter must be less than about10 French in size in order to safely access the interior of humanorgans, such as the heart. A catheter of a larger diameter could presentinsertion complications, clotting, and flow-blockage risks to thepatient. Also, larger diameter catheters are more difficult to bendthrough the arteries or veins by which access to an organ is obtained.On the other hand, piezoelectric transducers are limited to a minimumsize range by the ultrasound frequencies desired for imaging purposes.In the intracardiac imaging application, desired ultrasound frequenciesrange from 3 to 10 MHZ, and typically range between 5 and 7 MHZ. Inorder to be able to produce ultrasound within this frequency range, eachtransducer element must have a minimum dimension (length, width andheight) of approximately 0.2 square millimeters Further, the spacingbetween such elements is also governed by the range of imagingfrequencies employed, and the related side-lobe characteristics based onthe lateral sensitivity of the crystal configurations used. For example,a linear phased array imaging at 4.5 MHz to 7.5 MHz, could have a pitchof 0.2 mm. As a result of these two dimensional limitations (i.e.,catheter diameter and minimum transducer dimension), the onlyconfiguration possible for a phased array of piezoelectric transducersin an intracardiac catheter is a linear array aligned with the long axisof the catheter. A conventional intracardiac linear phased arrayultrasound imaging catheter is shown in FIG. 3.

A linear phased array ultrasound transducer 22 can only generate a twodimensional slice image spanning an angle of imaging 26 by steering theultrasound beam 260 up and down along (i.e., parallel to) the long axisof the array, as illustrated in FIG. 4. Consequently, a linear phasedarray ultrasound imaging catheter acquires a two-dimensional image withan image plane parallel to the long axis of the catheter. Since theemitted beam is not a pure line but a narrow conical beam (i.e., it hasa diameter), the image plane has a thickness T depending upon thedistance from the transducer. However, the ultrasound system recordsechoes as single returns with distance from the transducer 22 determinedfrom the time of arrival of each echo. Each echo may be represented as apixel on an ultrasound image frame. Due to the finite pulse width andthe height and width of the beam T, each pixel in the ultrasound imageframe represents a small volume. This volume is small as the typicalintracardiac ultrasound transducer emits a beam of sound that has anangular dispersion of approximately 0.5 mm at typical imaging distances.For this reason ultrasound pixels are sometimes referred to herein as“voxels” consistent with the use of that term in the ultrasound imagingarts. Even though the resulting ultrasound image frame represents a thinvolumetric slice, it is not a three-dimensional ultrasound image. Togenerate three-dimensional images, a number of generally contiguousultrasound image frames must be combined, such as by using methodsaccording to the various embodiments of the present invention.

For some diagnostic applications, it will be advantageous to generatethree dimensional images in order to view significant portions of anorgan at the same time. For example, in the heart, a two-dimensionalimage frame shows only a small cross-section of the heart at a time.Yet, the heart is a very complex three-dimensional structure. If thetransducer array is oriented to view a cross section of the heart thatcontains healthy tissue, the clinician may not detect adjacent tissuewhich is behaving abnormally or is diseased. Further, as the heartbeats, the surfaces of the ventricles and atria move in a complexfashion. Thus, it is difficult for a clinician to visualize the entireheart or comprehend how the various structures are moving throughout thecardiac cycle when the clinician is only able to view a single thinslice image at a time.

To overcome the limitations of this two-dimensional imaging capability,clinicians will typically rotate the catheter during an examination inorder to view slice images of different parts of the heart. By rotatingthe catheter back and forth, a clinician can scan the inside of theheart, much like swinging a flashlight back and forth to view a darkroom. While this procedure allows the clinician to see much of theheart, it is necessarily limiting in utility for at least three reasons.First, the clinician must rely upon memory to piece together the variousviews obtained in each of the two-dimensional slices in order tovisualize the three-dimensional structure of the heart. This proceduremay be facilitated in offline analysis when multiple adjacent images maybe displayed on a computer screen simultaneously. However, such methodshave limitations since it is difficult to visualize a three dimensionalimage of an organ with a shape as complex as the human heart. Also, theviewing perspective (i.e., position and orientation) of the imagingtransducer may change from image to image as the catheter is rotated.Second, the heart is a dynamic organ, moving and changing shape severaltimes per second. Consequently, as the ultrasound imaging transducer isrotated to a new viewing angle, it is imaging the heart at differentinstants in the cardiac cycle. The clinician may attempt to overcomethis disadvantage by slowly rotating the catheter so that multiple beatcycle images are observed at each rotational orientation. However, thisextends the examination procedure and further complicates theclinician's task by requiring visualization of the three dimensionalstructure which is changing shape constantly. Third, when the clinicianrotates the catheter, the position and angular orientation of thetransducer array may move in an unpredictable manner. For example,rotating the catheter may cause the transducer to shift laterally inposition and/or rotate upward or downward with respect to the previousviewing orientation. Also, forces from movement of the heart or bloodflow may cause the transducer array to move from image to image.Consequently, a clinician is unable to know whether changes in locationof imaged structures viewed in subsequent two-dimensional slicing imagesare the result of the shape of the heart structure or movement of thetransducer array with respect to the structure, or both.

As result of these difficulties, current intracardiac ultrasoundcatheter imaging systems have little if any ability to generatethree-dimensional images of the heart. Methods for correlatingultrasound images in time, particularly with respect to the cardiaccycle, have been disclosed in U.S. Pat. No. 5,722,403 and U.S. PatentPublication No. 2005/0080336, which are both hereby incorporated byreference in their entirety. Nevertheless, additional methods are neededfor accurately stitching together two-dimensional ultrasound images inorder to render an accurate three-dimensional representation of cardiacstructures. While cardiac imaging represents a particularly urgent needfor image processing and combining methods, such methods could also beuseful in the examination of other organs. To address this need, thevarious embodiments enable the generation of three-dimensional imagesand four-dimensional (i.e., three-dimensional plus time) movies (alsoreferred to herein as “cines”) from a series of two-dimensionalultrasound image frames. The embodiments also enable the generation ofthree- and four-dimensional image datasets to enable virtual closeinspection of particular portions of the heart.

In order to generate such merged images from a dataset of ultrasoundimages, a number of technical challenges must be overcome. Before asequence of images can be combined, the images need to be correlated intime (particularly for a moving organ like the heart) and with respectto the transducer viewing perspective (e.g., its position/orientation).Additionally, the inherent variability in ultrasound image quality fromframe to frame due to noise, speckle and other phenomena needs to beprocessed out or otherwise accounted for. Finally, the rawtwo-dimensional images or the assembled three-dimensional image may needto be processed to identify or emphasize clinically significant detailsor features. These and other technical problems are addressed in thevarious image processing embodiments described herein.

To obtain a series of two-dimensional ultrasound image frames using asystem like that illustrated in FIG. 1, a sterile ultrasound imagingcatheter 20 may be introduced into the patient's body, such as bypercutaneous cannulation, and positioned so the transducer array 28 isat a desired location and orientation, such as guided by use offluoroscopy. The ultrasound unit 40 is initialized and images obtained.The position and orientation of the imaging transducer array may bedetermined with respect to a frame of reference. The frame of referencemay be with respect to the patient or the organ being imaged, withrespect to the examining table, with respect to the examining room, orwith respect to an arbitrary frame of reference, such as that of thefluoroscopy equipment or the examining table. Locating the imagingtransducer within a frame of reference facilitates combiningtwo-dimensional image frames obtained from the transducer by providing aknown location (i.e., the source) in each image. Apparatus, systems andmethods for locating the ultrasound imaging transducer within a patientare disclosed in U.S. patent application Ser. No. 11/610,357 entitled“Catheter Position Tracking for Intracardiac Catheters” filed Dec. 13,2006, and Ser. No. 11/610,386 entitled “Catheter Position TrackingMethods Using Fluoroscopy and Rotational Sensors” filed Dec. 13, 2006,both of which are incorporated herein by reference in their entirety. Itis noted that the step of determining the transducer position andorientation is not required for all embodiment methods, since somemethods are able to align and co-register images by recognizing commonstructure points.

With the ultrasound system setup and configured, and the location andorientation of the imaging transducer array determined and recorded, aseries of ultrasound image frames are obtained and recorded in thesystem. ECG signals may be recorded and stored in the system with theimage data, such as in a correlated dataset. Once a sufficient number oftwo-dimensional images are obtained at a particular position andorientation, the transducer array can be rotated (and/or moved) to a newviewing perspective and the process of determining/recording thetransducer position and orientation, obtaining/recording images and(optionally) recording ECG signals is repeated. Then, this process isrepeated over a number of viewing perspectives in order to obtain aseries of image frames spanning the region of the organ for which athree- or four-dimensional image or image database is desired.

Once the dataset of images, transducer locations and (optionally) ECGsignals have been obtained, the image processing methods described belowmay be employed. These methods may be performed offline (i.e., after theimages have been obtained) or in near real-time, (i.e., at the same timeimages are obtained), or a combination of in real time and offlineprocessing.

In concept, assembling a series of ultrasound images into athree-dimensional image sounds easy; however, several technicalchallenges must be overcome to generate diagnostically useful three- andfour-dimensional images from two-dimensional ultrasound image framesobtained from ultrasound imaging catheters. This is particularly truewhen the images are of the heart, and even more the case when the heartis diseased in which tissues may exhibit irregular, unpredictablemovement, such as may occur during fibrillation.

The technical challenges which must be overcome to generatediagnostically useful three- and four-dimensional images fromtwo-dimensional ultrasound images may be grouped in four categories.First, there is the challenge of determining the precise viewingperspective of the imaging transducer. In a beating heart deep within apatient, it is difficult to determine exactly from where an image wasobtained, which makes it difficult to determine how images should bepieced together. Second, there is the challenge of imaging structurewhich is constantly in motion in order to assemble an accurate image ofthe structure at a particular dynamic state. This is particularly achallenge when the organ is moving in a random or unpredictable manner,as may be the case in a diseased heart. Third, ultrasound imagingpresents unique image processing challenges due distortions and noiseinherent in the nature of ultrasound, how it interacts with tissue andblood, and how ultrasound echoes are processed into images. Fourth, inorder to use the three-dimensional images for precision diagnostic,treatment and surgical applications, imaged structure must be preciselylocated within a coordinate frame of reference useful for a medicalprocedure.

Determining the Imaging Perspective.

Turning to the first technical challenge, an ultrasound imaging systemmust be able to identify the viewing perspective of each image in orderto accurately assemble adjacent images. Without this information, it isnot possible to distinguish a shift in viewing perspective from acurvature of structure between adjacent image slices. While thischallenge is present in all situations where multiple images arecombined into a composite three-dimensional image, within a beatingheart the challenge is particularly difficult. This is due in large partto the fact that the ultrasound transducer catheter introduced into theheart via venal cannulation lies deep within the patient and far fromthe point of insertion. It is not possible to know the location andorientation of the imaging transducer array on the tip of the catheterby observing the location and orientation of the catheter handle.Additionally, the large volume of the heart chambers makes it difficultto identify the precise location of the transducer within a chamber(e.g., atrium, ventricle or vena cava). Further, the large volume ofheart chambers permits the transducer tip to assume a wide range oforientations (i.e., it is not constrained to a particular orientation aswould be the case of a transducer within a narrow vein or artery). Also,the rapid contractions of heart muscle and surges of blood throughchambers and flow channels may cause the transducer end of the catheterto move unpredictably.

To accurately locate an ultrasound transducer within an organ of apatient in order to generate composite ultrasound images, both thelocation of the transducer in three-dimensional space and thetransducer's orientation within three degrees of freedom (i.e., pitch,roll and yaw) must be known. As it is used herein, “position” generallyrefers to the location of at least one a point on the transducer arrayin three dimensional space, and “orientation” refers to the pitch, yawand roll (i.e., rotation about the long axis) orientations of thetransducer array about those three degrees of freedom. The location andorientation degrees of freedom are illustrated in FIG. 5. Referring toFIG. 5, the direction D represents the mid-line of the two-dimensionalimaging plane (shown in the plane of the image) which extends along aplane parallel to the length of the array 22 and perpendicular to theface of the sound emitting faces of the array elements. As shown, thetransducer tip is capable of being located in 3 dimensional space(represented by x′, y′, z′). Similarly, the base of the transducer canalso be located in space through x, y, z dimensions. Further, thetransducer can rotated through an angle of θ around its longitudinalaxis and oriented so the linear array is tilted up/down (inclination orpitch angle) and angled left/right or into/out of the page (yaw angle).

The position/orientation of the transducer array 22 in each of these sixdegrees of freedom needs to be accounted for in constructing andcombining images. Movement of the transducer array between oneultrasound image and the next through any one of the six degrees offreedom will result in significant changes in the viewing perspective.Thus, the positional measuring technique used with the ultrasoundimaging system must fix the transducer's location in all six degrees offreedom (i.e., determine the values for the X, Y, Z dimensions and threeorientation angles) within a short measurement time constant (i.e.,minimum time to measure a position) relative to the contraction time ofthe heart.

Just as constructing three-dimensional images from a series oftwo-dimensional images requires knowledge of the transducer arrayposition and orientation information for the six degrees of freedom,positional errors must be accounted for in each of these six dimensions.Errors in each dimension of position and orientation combine to yieldthe total positional error of each ultrasound image. Further, a combinedimage will have positional errors of features that are combinations ofthose of the component images. If the resulting total positional errorof the series of ultrasound image frames is too large, the combinationof the images will be fuzzy, unreliable or otherwise unusable fordiagnostic purposes. Such errors result from both the sensitivity of theposition sensing mechanism used and the cumulative integration andderivation errors involved in position/orientation calculations.

Imaging and Processing Images of a Moving Organ.

Generating an accurate three-dimensional image of the heart isespecially difficult because the organ is constantly in motion. Beyondthe difficulties of determining the imaging perspective discussed above,a particular two-dimensional slice image shows the shape of the heart atonly a brief instant in time. In the next instant, the heart will havechanged shape so the next image may show heart structure in a verydifferent configuration. To build up a three-dimensional image of theheart using two-dimensional slice images, the composite may be based onimages taken when the heart is in the same shape. When the heart isbeating normally in rhythmic fashion, its shape changes in a repeatingpattern which can be used to select two-dimensional slice images thatcan be combined to generate an accurate three-dimensional compositeimage. Also, for diagnostic purposes, it may be most desirable to obtainthree-dimensional images of the heart at many different phases withinthe heartbeat cycle, or a three-dimensional movie (referred to herein asa four-dimensional representation) of an entire cardiac cycle. Obtainingsuch three- and four-dimensional images from two-dimensional sliceimages taken over a span of time requires a large amount of imageprocessing.

A rapidly moving heart poses a challenge related to correlatingultrasound images in time due to the duration of each ultrasound imagescan and movement of the muscle between the images. Heart muscle maymove rapidly during the peak of contraction, such as during the QRScomplex portion of the ECG, illustrated in FIG. 6A, or during periods offibrillation, illustrated as region 601 in FIG. 6B. Due to physicallimitations, such as the speed of sound in blood, pulse repetition rateof the transducer, and the size of the heart itself, eachtwo-dimensional ultrasound image frame takes a minimum amount of time toaccomplish. If two images are taken during a period of rapid movement ofheart muscle (such as during region 601 in FIG. 6B), the muscle may movea measurable amount between the two ultrasound image frames. The resultmay be a blurring or apparent thickening of the moving structure if thetwo image frames are added, averaged or otherwise combined. Insituations of rapid heart movement, such as may occur in a diseasedheart exhibiting fibrillation, the difference in the time of the portionof the image at the beginning of an image scan to the end of the imagescan may be significant.

As explained above, the linear phased array transducer generates atwo-dimensional slice image by steering a series of millisecondultrasound pulse beams through a series of angles with respect to thetransducer array face, e.g., from −45 degrees to +45 degrees to theperpendicular. This scanning of the ultrasound beam is illustrated inFIG. 4. In a typical intracardiac linear phased array transducer system,each scan from the bottom of the image to the top spans approximatelyone-sixteenth ( 1/16) of a second. For these reasons, carefulconsideration needs to be given to how ultrasound images are identifiedin time (“time stamp”) and the inherent error associated with such timevalues. For example, the time stamp of an image may be assigned when theimaging scan begins (i.e., when the first pulse is generated), in themiddle of the imaging scan (i.e., when the ultrasound pulse is directedperpendicular to the transducer array), at the end of the imaging scan(i.e., after the last echo of the last scan pulse is received), or atsome point in between.

In addition to addressing the finite duration of each ultrasound imageframe, image processing must recognize and accommodate the fact thatdifferent parts of the image, such as the outer edges of the image willhave larger time value errors (i.e., errors in the time value assignedto each portion of an image) than other portions of the image. Inultrasound imaging, a pixel time error is directly proportional to pixelposition error (related by the speed of sound in blood and tissue).Therefore, such time errors can become significant when assembling acomposite image of heart structure from many ultrasound images.

In addition to these image time errors inherent in each two-dimensionalimage frame, consideration must be given to timing errors introduced bythe use of ECG signals to correlate cardiac images. While the heartfollows a rhythmic and repeating pattern in response to electricalsignals picked up by ECG electrodes, there is some inherent variabilitybeat-to-beat, both in the timing of contractions against the ECG signaland in the movement of individual portions of the heart. In a diseasedheart, the heart muscle coordination with the ECG signal is poor,particularly in the atrium where electrical signals may not match thetissue motion. As a consequence, if the ECG signal is used to correlateor time-gate two-dimensional slice images in order to build up an imageset showing a three-dimensional portion of the heart, the correlationmethod must recognize and accommodate the timing and positional errorsdue to (1) differences between the heart's electrical activity andmechanical movements, (2) the finite duration of each imaging scan, and(3) the round trip duration of individual ultrasound pulses.

Consideration should also be given to the relationship between the speedof anatomical structure movement and the sensitivity of the ultrasoundimage. When heart muscle walls are moving rapidly with respect to theimaging duration (i.e., the amount of time required to acquire a singletwo-dimensional ultrasound image frame), the result may not yield a“bright” image of the moving structure. For example, if the ultrasoundimaging pulse has a pulse width ΔT (which is typically about 1/65^(th)of a second), the associated image may be recorded as occurring at T1,T1+ΔT, or any time in between. If the atrial wall being imaged is movingrapidly, its position at T1 may be removed from its position at T1+ΔT,and the resulting image may be blurred so that it is difficult toidentify edges or accurately measure the position of the vessel wall.Also, a moving structure will appear tilted as the structure movesduring the time the transducer array scans from the bottom of the imagesector to the top. Such blurring and distortion of moving structure mayneed to be considered in edge recognition algorithms used to recognizestructure in ultrasound images.

Challenges Inherent with Processing Ultrasound Images:

Generating an accurate three-dimensional image of the heart fromtwo-dimensional ultrasound slice images also requires solving a numberof problems caused by or inherent to two-dimensional phased arrayultrasound imaging. These problems include: distance-dependent “voxel”volume; signal attenuation with imaging distance; image noise and“speckle;” and multipath sound scattering creating pseudo-edges.

Distance-dependent Voxel Volume. Any image processing method whichcombines multiple images or recognizes structures using edge recognitiontechniques should account for the fact that the volume represented byeach “voxel” of information depends upon the distance from thetransducer array. This is due to the geometry of a phased array scan,wherein the center of the transducer forms the source point of each scanline, from which an evenly disperse angularly displaced set of scanlines form the scanned image. Thus, the distance between any two scanlines increases as a function of distance from the surface of thetransducer. This can further be extended to any rotational imageacquisition, where individual image slices, angularly displaced from acommon origin point (the center of the transducer), move further apartwith distance from the scan origin. The angle of resolution of thetransducer array depends upon the length of the array, the wavelength ofthe ultrasound pulse, and the angular sensitivity. A typicalintracardiac transducer array is approximately 13 mm long and theultrasound wavelength is approximate 0.3 mm in blood. The volumerepresented by a pixel, i.e., a “voxel”, of ultrasound informationincreases with distance from the transducer array. For example, in atypical intracardiac ultrasound imaging application, the volume of animage voxel at a typical maximum imaging depth of 16 cm is 60% greaterthan the volume of an image pixel at the typical minimum imaging depthof 10 cm.

This distance-dependent voxel volume relationship can result in imagedistortion when various two-dimensional images are combined into athree-dimensional composite image. Also, if images of structure takenfrom different imaging depths are combined, the structure images willneed processing to take into account the different volumetricresolutions of pixels in the respective images. Further, imagingprocessing methods which rely upon edge recognition algorithms may needto account for imaging distance in order to have consistent edgeresolving capabilities over the entire image.

Signal Attenuation. Attenuation of ultrasound in tissue and blood limitsthe imaging depth achievable with the ultrasound frequencies and powerlevel used in intracardiac imaging. As it passes through tissue,ultrasound is scattered and absorbed by tissue and blood, reducing theamount of energy returning to the transducer array. Consequently, theamount of ultrasound returning to the transducers from structure at thefar edge of the ultrasound image will be much lower than from structurecloser to the transducer array. As a result, there will by a lowersignal-to-noise ratio in image pixels near the edge of the imaging depththan closer to the transducer array, and thus lower imaging sensitivity.Image processing methods which do not take this phenomenon into accountmay not recognize structure near the maximum imaging depth, or mayinterpret noise as structure. Also, structures near the maximum imagingdepth may appear ill-defined or insubstantial due to signal attenuationeven though the structure is thick (e.g., a ventricular wall). If thegain on received ultrasound is increased in order to identify structurenear the maximum imaging depth, the result may be increased noise in thenear field portion of the image.

Image Noise and “Speckle”. In addition to the foregoing image processingchallenges, an ultrasound imaging system must also deal with noise inultrasound image frames. Three sources of noise in ultrasound imagesshould be accounted for. First, electronic noise is caused by ambientelectromagnetic interference (EMI). A typical “cath lab” has numerousEMI emitters in the form of electronic equipment, computers, fluoroscopyequipment, power and lighting sources. EMI noise can be significantbecause the electrical signals received from ultrasound transducers arevery weak and must be amplified significantly to generate an image.

The second source of noise in ultrasound images are due to random orenhanced echoes of ultrasound pulses which can appear as random pixelsand bright spots, referred to as speckle. Some specular reflections arecaused by constructive interference of sound waves reflected off ofclosely spaced structure layers (such as cell layers separated by aboutone-half the ultrasound wavelength). Such structure-related speckle canbe used for locating and tracking structure, such as disclosed in U.S.patent application Ser. No. 11/610,888, entitled “Method And System ForEstimating Cardiac Ejection Volume And Placing Pacemaker ElectrodesUsing Speckle Tracking,” which is hereby incorporated by reference inits entirety. Other sources of speckle are random in nature, appearingas light pixels in one image that are not present in the next.

A third source of noise in intracardiac ultrasound images is caused byechoes from ultrasound beam side lobes and ultrasound scattering off ofmultiple structures and blood such that sound follows an indirect pathbefore returning to the transducer array. Referred to herein as“multipath” scattering, such sound waves approach the transducer arrayfrom an angle different from the original ultrasound pulse and mayarrive delayed in time due to the longer path that the sound traveled.As illustrated in FIG. 7, one source of multipath scattering is causedby ultrasound from the main beam 261 being refracted and/or scattered bytissue and blood so that it follows indirect paths 268 back to thetransducer 22, unlike normal ultrasound echoes which follow direct paths262, 263, 264, 265 back to the transducer 22. Since ultrasound followingan indirect path 268 travels further before returning to the transducer,the ultrasound system may interpret multipath sound as being reflectedfrom more distant structure. Additional multipath interference can arisefrom echoes of beam side lobes 266. While the linear phased arraytransducer generates a focused primary beam 261, some of the ultrasoundenergy is emitted at an angle to the primary beam in the form of sidelobes 266. Echoes 267 of side lobe ultrasound from structure will returnto the transducer, and may be interpreted as being reflected fromstructure along the axis of the primary beam 261. Multipath soundscattering thus can be a source of noise and false signals in ultrasoundimages.

Correlating ultrasound images to a usable frame of reference. For manydiagnostic purposes it is important to correlate the ultrasound image toa frame of reference so that the position of imaged structure can belocated with respect to the patient's body, the examining table, otherexamination or surgical equipment or another frame of reference. Forexample, telerobotic, image guided or arthroscopic surgical or therapysystems are capable of positioning a surgical or therapy instrument at aprecise point within a patient. To guide such equipment to a particularpoint for a procedure, the ultrasound image results need to becorrelated (registered) to the positioning frame of reference of thesurgical or therapy equipment. As discussed above, the position of theintracardiac ultrasound image transducer within the patient is difficultto determine. This challenge is magnified when the ultrasound imagesmust be precisely located within an external coordinate frame ofreference of a precise machine-guided instrument system.

The foregoing imaging and image processing challenges may be overcomeusing one or more of the following image generation and image processingmethod embodiments.

Determining the Transducer Position/Orientation.

A number of methods for precisely locating a catheter within a patienthave been proposed and developed. Example methods and equipment for suchpurposes are disclosed in U.S. Pat. Nos. 5,515,853 and 6,192,266.Additional example methods and equipment for such purposes are disclosedin the following U.S. Patent applications: Ser. No. 10/994,424,published as US 2006-0122514 A1, entitled “Method And Apparatus ForLocalizing An Ultrasound Catheter;” Ser. No. 11/610,357, filed Dec. 13,2006, entitled “Catheter Position Tracking for Intracardiac Catheters;”and Ser. No. 11/610,386, filed Dec. 13, 2006, entitled “CatheterPosition Tracking Methods Using Fluoroscopy and Rotation Sensors.” Eachof the aforementioned patents and patent applications are incorporatedherein by reference in their entirety.

The frequency of baseline transducer position measurements androtational position estimations (using any of the techniques describedin the foregoing U.S. patent applications incorporated by reference) aswell as the imaging frame rate need to be sufficiently high to providethe degree of resolution required by the particular diagnosticobjective. Further, the positional (baseline measurements plusinstantaneous displacement estimates) and rotational measurements andimaging scans may need to be timed so that all three of thesemeasurements/estimations occur within an acceptable time-span ortime-correlation error band to permit clinically acceptablethree-dimensional image generation. This latter concern may arisebecause the duration required for recording each position/orientationmeasurement and/or image scan may be different. As a result, there willbe errors (i.e., degree of uncertainty) in the time at which eachposition measurement is obtained and thus a position/orientation errorassociated with each ultrasound image. If such errors are not properlymanaged or otherwise taken into account during image processing, theresult may be a blurring of the generated three-dimensional images.

In an embodiment illustrated in FIG. 8, the limitations and errors ofmeasuring the transducer position and orientation by external means areobviated by processing the obtained images to estimate the transducerorientation from the images themselves. In this embodiment, image framesare processed to determine the position and orientation of thetransducer array with respect to known or recognized structures in theimage. Ultrasound image frames may be accessed from memory, step 801, orreceived directly from the ultrasound imaging system step 802. Imagesfrom either source are processed using edge-detecting image processingalgorithms, step 803, to recognize the surfaces of structures within theheart and store positional information for the recognized surfaces.

Edge-detecting algorithms determine the edge of a tissue structure bynoting a sudden rise (for leading edge) or fall (for trailing edge) inbrightness across a short distance. Heart tissue structures may notexhibit a single bright edge in ultrasound, and instead exhibit areflectivity gradient across the endothelial layer and other tissuelayers between the outer edge and the inner portion of the structure. Asa result, determining where the structure edge lies may require imageprocessing using a number of alternative or complementary algorithms todecide where a structure edge exists or where the edge should bedetermined. For example, for some structures, the processor may comparethe brightness (i.e., amount of reflected ultrasound) versus distancealong each ultrasound beam to detect a sudden rise over a short distance(e.g., large Δbrightness/Δdistance) and determine that an edge exists ata pixel where this rate of change exceeds a threshold.

As another example, which may be used alternatively or in addition tothe first example, the processor may first determine that a structureexists in a portion of an image frame and then determine the edge ofthat structure. In this second example method, the processor may firstrecognize that a structure exists along a portion by noting that theportion has a higher average brightness than other darker portions wherethere is no structure (i.e., the darker portions contain blood whichreflects less ultrasound). Once a structure has been recognized, theprocessor can compare brightness values of pixels along a line segmentspanning the determined structure and a portion of the darker region todetermine where the edge of the structure exists. This determination maybe based upon a number of criteria, such as for example: (a) a point ofinflexion of the brightness versus distance along this line segment, (b)a pixel with an intermediate brightness value (such as the pixel closestto the median brightness value, or the pixel having a brightness valuethat is some fraction of the difference between the average bright andaverage dark region pixels) between the darker region and the brighterregion, (c) the first pixel in the segment with an increased brightnessover the darker region, or (d) the first pixel in the segment that hasbrightness value approximately equal to or greater than the averagebrightness in the bright region. These four example criteria will setthe edge of the structure close to the outer portion of the structure(the true edge), toward the middle of the edge region, or at the inneredge where the structure's inner tissue begins.

Other decisional criteria may also be employed, such as an iterative andstatistical analysis of all edge regions in the image frame to selecteda criterion that provides a smooth (i.e., non-discontinuous) edge acrossall radians in the image frame.

When the processor recognizes that a structure exists at a particularlocation, the image data can be stored as the positional (X, Y, Z)measurements of the edge location (e.g., as a three-dimensional vector),rather than as image pixel data. This process can thus generate athree-dimensional structure surface dataset. When the back edge isdetermined by the processor and the positional measurements stored asanother structure surface dataset, the area between the inner and outersurfaces defines the area of the structure in the cross sectional image.The processor may thus define and store a volumetric structure dataset.

With recognized surface and structure datasets from each image framestored in memory, the processor then can measure the distance and angleto the recognized structure, as well as structural dimensions, and fromsuch measurements deduce the location of the transducer at the time theultrasound image frame was obtained, step 804. Recognizing the structureand using the process to determine the imaging perspective may be bestaccomplished using easily recognized structure which undergoes littlemovement during the cardiac cycle. For example, the septum is generallyplanar and remains in a relatively stable position with respect to therest of the heart during the cardiac cycle. Also, the septum is thickcompared to other heart structures, and thus may be recognized by theprocessor based upon the distance between the inner and outer edgescompared to a table or range of typical septum thickness values orcompared to other heart structures in the image. Also, in certainviewing positions, such as when the transducer is in the right ventricleoriented toward the left ventricle as shown in FIG. 2, the septum may berecognized as being the closest structure to the transducer. The septummay also be imaged using fluoroscopy (albeit faintly) in order toprovide localizing information using an external sensor. In manyultrasound viewing perspectives used in cardiac diagnostic procedures,the septum will be included (at least partially) within the ultrasoundimage, and therefore useful as a common reference structure within twodimensional ultrasound image frames. Other useful structures that may berecognized by the processor for the purpose of self registration ofultrasound images include portions of the bicuspid and mitral valves andthe interior surfaces of the ventricles.

Once the heart structure has been recognized and its surface and/orvolumetric datasets determined, the processor can compare thisinformation to an anatomical model of the heart, step 805. Any of anumber of known algorithms for comparing measured dimensions tothree-dimensional digital models may be used to conduct this comparison.Then, using the comparison, the processor can back calculate from thiscomparison to estimate the location and orientation of the transducerthat would yield the obtained image of the recognized structure at themeasured distance and orientation, step 806. Any of a number of knownalgorithms may be used to estimate the transducer viewing perspective inthis manner. The estimated transducer array position and orientationinformation may then be stored in memory with the ultrasound image framefor further processing, such as the generation of three-dimensionalimages.

Additionally, the measured position of recognized structure or theestimated transducer array position and orientation information can becorrelated to an external frame of reference so the image data can becorrelated to the patient's body, step 807. Any of a number of methodsmay be used to correlate the imaged structure or transducer arraypositional information to an external reference frame. For example, thetransducer array may be imaged with fluoroscopy thereby providing adirect measurement of the transducer array in the two coordinate framesof reference. Similarly, the septum may be recognized and located inboth fluoroscopy (in the external reference frame) and the ultrasoundimage surface or volumetric datasets (in the internal reference frame).Once a structure or the transducer has been located in the two referenceframes, the processor can compute a coordinate transformation that willcorrelate the rest of the ultrasound image to the external referenceframe.

Self registration methods may also make use of catheter position ororientation information which is not expected to change with movementsof the heart. For example, the angular rotation of the transducer arrayabout its long axis is unlikely to be affected by muscular contractionsor blood flow around the catheter. Therefore, the rotational angle ofthe transducer array measured by a variety of methods (see, e.g., U.S.patent application Ser. Nos. 11/610,357 and 11/610,386 previouslyincorporated by reference) which may be combined with the selfregistration information obtained by recognizing a selected heartstructure (e.g., the septum) to provide a more accurate estimation ofthe viewing perspective. For example, the septum will appear as a thickline in a two dimensional ultrasound image slice when the left ventricleis imaged from a position within the right ventricle. While the linearimage of the septum will provide distance and angular information thatcan be used to estimate the transducer's distance from the septum, theremay be ambiguity as to which part (edge, middle, top or bottom) of theseptum is being imaged. By knowing the angular rotation of thetransducer array, the processor can better estimate which portion of theseptum is being imaged.

Self registration techniques can also use operator inputs, such as userdesignation of portions of the ultrasound image to select and identifystructure to be used for registration purposes. Further, selfregistration techniques may use shape recognition algorithms toautomatically recognize structures based on their shape, size, soniccharacteristics, or other measurable characteristics.

In a user-designated structure embodiment, a clinician can point to aparticular structure within a displayed ultrasound image, such as byusing a pointing device like a mouse or light pen, to select anddesignate a particular portion of the ultrasound image to be used forimage registration purposes. For example, the clinician may indicate theseptum within an ultrasound image such as by touching the correspondingportion of the image with a light sensitive pen, or clicking a mousebutton when the cursor is positioned on the selected structure. Once thestructure has been selected, the clinician can then inform theprocessor, such as by means of a keyboard entry, that the designatedstructure should be recognized and used for purposes of aligningultrasound images using self registration. The clinician may alsoidentify the selected structure by a keyboard entry of information sothe processor can look up that structure by name in thethree-dimensional model of the heart. With this information, theprocessor can use edge recognition techniques to locate the samestructure in other ultrasound images in the image database or receivedsubsequently from the ultrasound system. This method allows theclinician to select structures which are likely to be present in allimages to be obtained, relatively stable in all images and have a knownor measurable position with respect to the portion of the heart beingexamined. For example, the clinician may select the septum for thispurpose because its position relative to an imaged ventricle is known.

In the embodiment employing shape recognition to select structures foruse in self registration of images, the processor can be programmed withdigital three-dimensional anatomical models of an average heart. Theprocessor then can be programmed to compare recognized structures (i.e.,structures identified within an ultrasound such as by edge recognitionalgorithms) to the stored anatomical models in order to determine ifthere is a near match to a particular structure model. If there is amatch, the processor then can use the anatomical model to estimate thelocation and orientation of the transducer array by working backwardfrom the model using the position and orientation of the recognizedstructure within the ultrasound image. In this process embodiment, theprocessor may first process an ultrasound image to recognize structuresusing edge recognition techniques, steps 803, 804. Having recognizedstructures, the processor may then compare the shape and characteristicsof those structures to the three-dimensional model dataset stored inmemory, step 805. If a match is determined within a set margin of error,the processor may then estimate the transducer array viewing positionand orientation based on the distance to the recognized structure andthe angle of the recognized feature in the ultrasound image withreference to the three dimensional heart structure model, step 806. Thisprocess may also be performed iteratively wherein an initial transducerposition/orientation is estimated, steps 803-806, after which theprocessor may scale the three dimensional structural model to reflectthe expected size of structure when viewed from that perspective. Thenthe steps of pattern matching the structure to the modeled structure,805, and back calculating the transducer array position/orientation,806, may be performed using the updated match data. Such an iterativeprocess may increase the position estimation accuracy in two, three ormore iterations.

As described above, transducer array rotational angle informationobtained by other mechanisms may be used in combination with imagerecognition techniques to more accurately determine the transducerposition and orientation at the time each ultrasound image is obtained.For example, information regarding the rotational angle of thetransducer array may be used by the processor to compare a particularrecognized structure (e.g., the septum) to a portion of the digitalthree-dimensional structure model of the heart.

In addition to the above autonomous and clinician assisted methods forrecognizing imaged structure, the processor may use learning systemalgorithms such as a neural network application to learn from clinicianinputs and corrections. A neural network system may learn fromdetermining edge locations in a series of ultrasound image frames, suchas from statistical analysis edge discontinuities within frames andbetween frames, in order to improve the criteria used for recognizingstructure. Also such a system may learn from clinician corrections tobetter recognize particular structures, like valves and the septum, inultrasound images, particularly for a specific patient whose heartstructures may deviate from an ideal three-dimensional model.

Solutions for Imaging the Moving Heart:

Normally, a healthy heart changes shape, contracting and expanding, in arhythmic and repeating manner throughout the cardiac cycle. The cardiaccycle is characterized by the ECG signal which is also rhythmic andrepeating. The expectation that the heart's shape changes in arhythmically and repeating manner has been relied upon in previouslydisclosed methods for assembling three- and four-dimensional images ofthe heart from a series of ultrasound images. For example, use of theECG signal to correlate a sequence of cardiac images is disclosed inU.S. Pat. No. 5,722,403 and U.S. Patent Publication No. 2005/0080336which have been previously incorporated by reference. A method oftime-gating a medical image to phases of the cardiac cycle according toan embodiment is shown in the flowchart of FIG. 9. This method may beperformed using the medical imaging system illustrated in FIG. 1.

In an embodiment of the medical imaging system illustrated in FIG. 1, anexternal electrocardiogram (ECG) unit 60 may be connected to theultrasound unit 40 through an ECG communications interface 62, with thesignals sent through interface 62 used to synchronize ultrasound imagingwith the heartbeat of the patient. For example, a sequence of images maybe associated with a sequence of phases of a cardiac cycle, or imagesmay be captured only at a specified phase of the cardiac cycle. The ECGsignal may be monitored by the processor (e.g., microcomputer 41) toorchestrate the operation and timing of the signal generator 46 in orderto image the heart at particular phases of the cardiac cycle.

Referring to FIG. 9, in step 210 a trigger signal of a timing ofinterest is obtained, such as from the ECG signal. In this regard, a“timing of interest” refers to any information corresponding to thedisplay and/or analysis of images in accordance with a user'srequirement based in whole or in part on a time of occurrence. By way ofexample, the trigger signal may comprise a periodic feature or wavewithin an ECG trace which may be a periodic or non-periodic signalcorresponding to a physiological condition (e.g., a signal generated byan ECG probe sensing intra-cardiac electrical activity, a pacingstimulation signal, etc.), a user selection-based signal (e.g., a userselected time point in a normal or abnormal ECG trace) or other triggersignal of a timing of interest. In a normal healthy heart, the QRScomplex, or more narrowly a portion of the R wave (e.g., as thetransition from the Q wave to the steeply rising signal of the R wave),may serve as a reliable triggering signal that is easy for the systemprocessor to recognize. Further, the trigger signal may be based upon arecognizable feature in the ECG trace, such as the transition to the Rwave, plus a time delay. This time delay then can be increased so that aseries of trigger signals spans a complete heartbeat cycle, with eachtime-delay trigger signal corresponding to a like portion of the ECGtrace. By using such a time-delayed trigger signal, a series ofultrasound images may be obtained (or selected from memory)corresponding to like portions of the ECG trace which, presumably, willcorrespond to like or similar structural shapes or configurations.

As a further example, the trigger signal may be a complex andintermittently recurring ECG wave form such as may appear in a diseasedheart which requires analysis of ECG signal patterns to determine thepresence of the complex signal (e.g., analysis of an ECG signal todetermine the presence of an irregularity to be used as the triggersignal).

In step 220, a plurality of two-dimensional ultrasound image frames of aportion of the heart are obtained and stored. This step 220 may beperformed prior to, concurrent with, or after step 210.

In step 230, the ultrasound imaging system correlates or time-gates(e.g., synchronizes) the plurality of two-dimensional ultrasound imageframes obtained in step 220 with the trigger signal obtained in step210. In an embodiment, this correlating or time-gating in step 230involves using the trigger signal obtained in step 210 to triggergeneration of the plurality of two-dimensional ultrasound image framesobtained in step 220 (i.e., each image is obtained only when aparticular trigger signal is sensed). By way of example, the ultrasoundequipment 40 (FIG. 1) may generate ultrasound pulses correlated to thetrigger signal such that an ultrasound image scan generated each time(or a set time after) a periodic trigger signal is received. In thismanner, the obtained trigger signal is said to “time-gate” the pluralityof images, because the plurality of images are obtained (i.e., “gated”)in accordance with a timing of interest. The result will be a series ofimages of heart structure (e.g., heart wall, heart valve, etc.) at thesame point in the cardiac cycle (within the timing errors discussedherein). If the imaged heart structure cycles through repetitivemotions, the series of time-gated image frames may be combined (e.g.,added or averaged) to enhance the clarity, or compared to identifyirregular motions. By sequentially varying a time lag following aparticular (e.g., easily recognizable) timing event (such as the Qwave), a series of time-gated image sequences may be obtained of thestructure at each of a number of intervals within the heartbeat cycle.Thus, the result may be a “movie” or “motion display” of an averageheart cycle.

Alternatively, the images obtained in step 220 may be stored along withor indexed by associated timing information. Such “timing information”may be any information that can be used to correlate the plurality ofimages with a timing of interest with respect to the dynamic conditionof the heart. Such timing information may include, for example, the timethat a particular ultrasound image frame is obtained, such as recordingthat a particular image was obtained at time “T” (plus or minus thetiming errors discussed above) that also is tied to the recorded ECGdata. Alternatively, the timing information may relate to acorresponding physiological condition, such as recording the ECG signalmeasured at the moment each ultrasound image slice is obtained. Further,the timing information may be relative to a recorded physiologicalcondition, such as recording the offset from (time since the occurrenceof) a particular wave in the ECG signal, such as time after Q wave orthe rising edge or peak of the R wave in the QRS complex. In thismanner, time-gating in step 230 may comprise utilizing the triggersignal obtained in step 210 to retrieve stored image framescorresponding to like portions of the ECG trace (e.g., images taken at atiming of interest) from a database of stored ultrasound image framespreviously obtained in step 220 and stored by the imaging systemworkstation. The retrieved images may then be combined or otherwiseprocessed as herein described.

Since time correlated or time-gated two-dimensional ultrasound imageframes can be correlated to particular three-dimensional states of theheart, a series of such image frames can be used in image processingmethods described herein to generate two-dimensional, three-dimensionaland four-dimensional composite images. For example, several images fromthe same viewing perspective (transducer position and orientation) canbe processed to yield composite images with improved image clarity,resolution, and/or reduce noise. As another example, several images atthe same correlated time within the cardiac cycle but at differentviewing orientations can be combined to generate a three-dimensionalcomposite image.

In a particular example, ultrasound image frames may be taken during theperiod of relative rest and relatively slow movement that a healthyheart exhibits between beats, illustrated in FIG. 6A as shaded regions600 of the ECG. During these intra-beat periods, several ultrasoundimages can be obtained of the heart in the same basic shape. Since theheart is moving little during this intra-beat period 600, many of theimage distortion and correlation problems caused by movement of thetransducer and/or heart tissue described herein can be minimized oravoided.

While these two embodiment methods are limited to imaging the heartduring diastole, the result may be an accurate three-dimensional view ofthe heart that can be diagnostically useful. These embodiment methodsare also limited to images taken during times of normal heartbeat, whenthe diastole phase can be recognized from the ECG signal and the heartrepeatedly returns to the same shape during moments of rest.

As another embodiment, a long series of images at various transducerorientation angles may be obtained and stored in the processor's memoryalong with data on the position and orientation of the transducer arrayat the time of each image, so that the processor can use the storedimages and transducer location/orientation data to build up approximatethree-dimensional images. In this example, the processor may use imageprocessing methods that recognize similar patterns to select and stitchtogether images that have consistent edge shapes. This method ispremised on an assumption that even in atrial fibrillation the heartreturns to certain similar shapes, which can be recognized in a seriesof images by selecting images that can be smoothly stitched together.

The foregoing embodiment methods that enable a three-dimensionalreconstruction of ultrasound images of the heart are best suited to aheart with a rhythmic cardiac cycle. However, generating athree-dimensional reconstruction of the heart may be particularly usefulwhen the patient is in atrial fibrillation, flutter, tachycardia, ordysynchrony since causes of such conditions may be deduced fromultrasound images. In such situations, the heart is flexing in irregularand unpredictable patterns that may be disjoint from the ECG patterns.In such conditions, methods that use normal ECG signals to assist informing a three-dimensional image may be infeasible since the positionof the heart walls may be unpredictable or the ECG pattern may beerratic and random.

In cases where abnormal heart conditions exist, such as rhythmabnormalities, and where such abnormality is atrial fibrillation inparticular, periods of mechanical inactivity may be brief or evenabsent. In such situations, multiple images may need to be acquired andprocessed using statistical methods to reduce the overall spatial errorin the estimation of composite three-dimensional images. For example,multiple images obtained during fibrillation may be combined using aweighted averaging method wherein locations (i.e., echo locations) thatappear in a majority of images within a given locale are given moreweight while spots that appear in a minority of images are given lessweight. Since during fibrillation the walls of the vessel (atria orventricle) may quiver about an average shape, this averaging method mayprovide an image of the average shape. In this embodiment method,various averaging and estimating techniques can be used to generate aseries of composite two-dimensional ultrasound images of the heart thatcan then be used to generate a three-dimensional image of the heart.

In another embodiment, the averaging technique uses statistical analysisof the atrial fibrillation ECG pattern in combination with edgedetection analysis of ultrasound images to correlate heart wall motionor position to patterns in the ECG signal. In this analysis, theprocessor statistically analyzes the position of structure andassociated patterns in the ECG signal. If a particular structureposition occurs in conjunction with a particular ECG pattern in astatistically significant number of images (such as one or two times thestandard deviation of position-to-ECG-pattern measurements), that ECGpattern may then be used to initiate ultrasound imaging or, inpost-procedure processing, to select ultrasound images for processing.Once sufficient ECG and image data has been obtained so the processorcan recognize correlations between ECG patterns and heart structurepositions in ultrasound images, the processor can image the heart atselected times or select images in post-procedure processing that willshow the heart in the same (or nearly the same) configuration based uponthe ECG signal (e.g., by using statistical analysis). Then, once aseries of two-dimensional ultrasound images are obtained or selectedbased upon analysis of the ECG signal, they can be processed andcombined using the further processing embodiment methods disclosedherein.

Under such conditions of irregular heartbeat, three-dimensionalreconstruction of ultrasound images may be accomplished by taking rapidultrasound image scans over small regions by quickly imaging androtating the transducer array between image scans. A few two-dimensionalultrasound images taken closely together in time may then be combined torender a three-dimensional image through a narrow angle of rotation.This embodiment method may be most effective when used in combinationwith an ultrasound imaging catheter capable of rapid rotation betweenimage scans, such as in U.S. patent Ser. No. 11/764,194 entitled“Oscillating Phased-Array Ultrasound Imaging Catheter System” and filedJun. 16, 2007, the contents of which are incorporated herein byreference in their entirety.

In a diseased heart, the contractions of various heart muscles may bequite rapid, out of phase, disorganized or chaotic, such that the shapeof the heart does not return to the same configuration or rest on aregular or repeating basis. Such periods of fibrillation may beidentified in the ECG trace as illustrated in the shaded portion 601 ofFIG. 6B. Also, episodes of such erratic heartbeats may occursporadically so it can be difficult to obtain images at a consistentpoint during such erratic heartbeats during the normal duration of acardiac catheterization procedure. Under such circumstances, it may notbe possible to image the heart in particular configurations, in whichcase the image processing methods described herein may be used to removethe blurring effect of rapid structure movement and correlate adjacenttwo-dimensional ultrasound image slices into a three dimensionalrepresentation of the heart.

In an alternative embodiment useful in common medical proceduresinvolving intracardiac ultrasound imaging, the signal of interest is apacing pulse applied to the heart such as by a pacemaker lead. When apacemaker is implanted in the heart of a patient, the clinician maystimulate (“pace”) the heart at a number of locations to identify alocation which exhibits the best physiological response to the pacingpulse. Once a pacing site is located, the clinician may adjust thetiming of the pacing pulses with respect to the heart cycle to determinethe phase lag or lead which results in the best physiological responseto the pacing pulse. This procedure may be conducted under theobservation of intracardiac ultrasound imaging to assist the clinicianin locating the electrode as well as observing the physiologicalresponse of the heart. In this embodiment, the pacing stimulation isused as the timing signal, which will be detected in the ECG signal, totime-gate ultrasound image frames as described above. Using thisembodiment in combination with the image processing methods describedherein, the clinician can obtain image frames which characterize theheart during pacing stimulation, both at the point of stimulation and atselected time intervals following stimulation.

In an alternative of this embodiment, the stimulation pulse may beapplied by the clinician in order to initiate or terminate fibrillation.In such a procedure, the stimulation is used to induce the heart toenter into a condition that is being diagnosed, such as episodicfibrillation, or to terminate an irregular heart beat. In such cases thestimulation pulse can be used as a triggering signal to obtain (orrecall from memory) ultrasound image frames to image the heart as itundergoes a transition into or out of a fibrillation or other irregularheart beat condition.

Methods for Processing Images of the Moving Heart:

Ultrasound image frames taken of the heart, particularly during periodsof rapid movement, can be processed according to the followingembodiment methods in order to overcome the challenges described above.

In an embodiment method illustrated in FIG. 10, many of the challengesof processing and combining ultrasound images of the moving heart aremanaged by using statistical processing of spatial data derived fromimages instead of statistical analysis of image data itself. If imagedata from several ultrasound images of a moving organ are statisticallyanalyzed, the result can be a blurring of the image. While statisticalanalysis of individual pixels as described above can remove random noisefeatures, such as speckle and electronic noise, the same analysis canresult in composite images with blurred edges as movement of tissue fromframe to frame will be statistically combined with image location errorsand other ultrasound image uncertainties. To avoid this effect, theembodiment method first recognizes the location of structure anddetermines its spatial coordinates, and then statistically analyses thespatial coordinate data to of recognized structure in multiple imageframes determine the most likely location of the structure in acomposite image.

Referring to FIG. 10, in this embodiment method, ultrasound image framedata may be retrieved from memory (e.g., from image frames stored in thememory of the ultrasound imaging system or in a database), step 1001, orreceived directly (i.e., in real time) from the ultrasound imagingsystem, step 1002. Individual ultrasound image frames are processed by aprogrammable processor, such as the workstation of the ultrasoundimaging system or an image analysis workstation, to detect structuresusing edge-detecting image processing algorithms, step 1003.

As discussed above with reference to FIG. 8, edge-detecting imageprocessing algorithms determine the edges of structures by noting asudden change in pixel brightness along image radians. For example, forsome structures, the processor may compare the brightness (i.e., amountof reflected ultrasound) versus distance along each ultrasound beam todetect a sudden rise over a short distance (e.g., largeΔbrightness/Δdistance) and determine that an edge exists at a pixelwhere this rate of change exceeds a threshold. As another example, whichmay be used alternatively or in addition to the first example, theprocessor may first determine that a structure exists in a portion of animage frame and then determine the edge of that structure. In thissecond example method, the processor may first recognize that astructure exists along a portion of a radian by noting that the portionhas a higher average brightness than other darker portions where thereis no structure (i.e., the darker portions contain blood which reflectsless ultrasound). Once a structure has been recognized, the processorcan compare brightness values of pixels along a line segment spanningthe determined structure and a portion of the darker region to determinewhere the edge of the structure exists. This determination may be basedupon a number of criteria, such as for example: (a) a point of inflexionof the brightness versus distance along this line segment, (b) a pixelwith an intermediate brightness value (such as the pixel closest to themedian brightness value, or the pixel having a brightness value that issome fraction of the difference between the average bright and averagedark region pixels) between the darker region and the brighter region,(c) the first pixel in the segment with an increased brightness over thedarker region, or (d) the first pixel in the segment that has brightnessvalue approximately equal to or greater than the average brightness inthe bright region.

When the processor recognizes that a structure edge exists at aparticular location, the coordinates of the recognized structure arestored as the positional measurements (e.g., X, Y, Z coordinates) of theedge locations or as vector or tensor coordinates (e.g., as athree-dimensional vector), step 1004. The positional measurements arefirst obtained with respect to the transducer array at the apex of theimage, such as radial angle transducer orientation angle, and distancevalues along each radian in the image frame. Optionally, the radialcoordinate values may be converted to rectilinear X-Y-Z coordinatesusing well known transformation algorithms. This process generates athree-dimensional spatial dataset defining the locations of structuresurfaces. If the back edge of a structure is imaged, the dataset willinclude the spatial coordinates of both the front and back edges of astructure, and a further processing algorithm can recognize that the twoedges define the same structure. This algorithm may be as simple as arule that closely spaced edges are associated with the same structure.The processor may thus define and store a volumetric structure datasetwhich spatially locates the structures recognized in the ultrasoundimage.

The process of recognizing structures, step 1003, and recording thespatial coordinates of recognized structures, step 1004, extractsstructure location data from ultrasound image data, thereby generatingan image dataset that can be statistically processed without compoundingthe errors and data uncertainties inherent in ultrasound images.

The processor can then statistically process structure spatial (i.e.,location) data across some or all of the image frames to determine anaverage, most likely or best fit location for the structure, step 1005.Statistically processing the spatial location of edges allows theprocessor to select for display a single representative location foreach structure edge from a multitude of images, with that singlerepresentative location being a close approximation to the nominalposition of the structure.

This process step is based upon the observation that heart structure isconstrained and thus moves (in and out, back and forth) about a centrallocation. For example, the ventricles pump blood by first expanding toincrease their volume and then contracting to decrease their volume, andas a result the ventricle walls first move outward and then inward inapproximately equal distances with respect to their nominal position. Asa first approximation, the central location of a constrained structureis the average of the spatial locations of the maximum and minimumvolume locations. As another example, heart tissue in fibrillation maymove rapidly but it does so about a nominal position. Thus, the nominalposition of a constrained structure that is moving can be estimated byaveraging (or other statistical calculation) the spatial coordinates ofthe structure across a number of image frames. The more images that arespatially averaged in this manner, the closer the average position willbe to the nominal position of the constrained structure. The estimatednominal position can then be used as the representative position forgenerating an image of the heart structure (referred to herein as arepresentative image).

Statistically processing the spatial location data for structures todetermine a representative location can provide better results thanother methods when the heart is moving rapidly and/or irregularly, asmay happen in a diseased heart undergoing fibrillation. In such cases,it may not be possible to “time-gate” ultrasound images or otherwiseselect images that correspond to the same shape or configuration. Inmany diagnostic situations, the shape of the heart during fibrillationis of greater importance than during normal heart beats. Also, duringfibrillation the heart structure may move too far between image framesor between transducer rotational orientations to permit the transducerto obtain ultrasound images of the structure in the same shape orconfiguration. This analysis method compensates for the effects of rapidmovement by calculating a single spatial representative location foreach image point on a structure that lies close to the center of motion.While the selected representative spatial locations do not reflect thelocation of structure at any particular point in time, the resultingimage of the structure is representative of the nominal structure of theheart and therefore useful for diagnostic purposes.

The processor may use a number of statistical analysis methods todetermine a single representative location for the structure from aseries of image frames. These methods include, for example, computing asimple average of the spatial location in all images, determining themean or median spatial location in all images, and computing a weightedaverage location which may incorporate statistical information orknowledge about the nature of the structure being imaged in the form ofweighting factors. For example, a weighted average analysis method thatcan be used to determine a single representative position may firstdetermining the average of all the spatial locations, and then compute aweighted average where each spatial location value is weighted basedupon its distance away from the average location.

In an optional step 1006 the processor may analyze the spatial locationdata among a number of images to determine statistical values of averagelocations and standard deviations about those locations, and thenidentify image frames in which the location values appear to beoutliers. Outlier locations may be identified as locations that are morethan one or two standard deviations from the average location, or othercriteria that may be set. The processor may then ignore the outlierimage location data in determining the average (or median) location ofeach structure. Thus, in this optional combination of steps 1005 and1006, the processor may determine the representative location of astructure as the average of locations excluding the outlier locations(e.g., locations exceeding 2 standard deviations from the average), themedian of locations excluding outlier locations, or some other locationdetermined statistically from the location data excluding outlierlocations.

The steps of selecting or obtaining an image frame, steps 1001, 1002,detecting edges in the images, step 1003, determining the spatiallocations of edges, step 1004, and determining the average (or otherstatistical measure) spatial location of the edges, step 1005 andoptional step 1006, are performed for all or a subset of the imageframes. In an embodiment, these process steps are performed for a subsetof image frames obtained from a particular transducer rotationalorientation. These steps are repeated for subsets of image framescorresponding to different transducer rotational orientation until acartoon rendered image frame has been generated for all transducerorientations spanning the range of transducer rotational orientations inorder to generate a spatial location dataset spanning the entire imagedvolume of the transducer array.

Working with a single image frame or the spatial location datasetcorresponding to a particular transducer rotational orientation, theprocessor can then generate a “cartoon” rendering of the structuredetected in a single two-dimensional image frame by linking together allof the average edge locations that form a line in the image, step 1007.In this step, the processor compares adjacent locations in the imageframe to determine if they are part of the same structure. Thisdetermination may be a simple comparison to evaluation criteria such ascloseness in space or relationship to a line of locations. If a seriesof spatial locations are determined to be the same structure, thelocations are linked by a line, or replaced by a line whose spatialdimensions (locations, vector coordinates, etc.) are stored in memoryinstead of individual location coordinates. Isolated point locations areleft as is since such edge detections could be a structure viewed edgeon such that it will appear as a line in a three-dimensional image butonly as a point in a two-dimensional slice image. Since the lines andsurfaces reflect average spatial locations rather than imagesthemselves, these generated features are referred to herein as a“cartoon rendering” of structure rather than a two- or three-dimensional“image.” This process step can be repeated for all image frames.

By cartoon rendering the average location of detected structure in imageframes corresponding to a particular transducer rotational orientation,a clean two-dimensional image of heart structure in its nominal position(as viewed in the particular transducer orientation) can be displayedfor the clinician, step 1008. This cartoon rendering display may havediagnostic benefits since it will cleanly reveal the nominal (e.g.,average) shapes of structures without interference of noise or speckleand without the blurring caused by rapid movement of heart tissue. Thus,this process provides an image of the nominal shape of the heart evenwhen it is in fibrillation.

The processor can also generate a three-dimensional cartoon rendering ofthe entire imaged volume by combining the cartoon rendered image framesfor all transducer rotational orientations or generating athree-dimensional cartoon rendering from the entire three-dimensionalspatial location dataset, step 1009. In this step, the processorcompares adjacent average image frames to determine which edge locationsare associated with the same structure and connect such edge locationswith a line or surface spanning the adjacent average image frames.Information regarding the transducer rotational orientation, which maybe stored or correlated with the image frames, is used by the processorto determine the spatial separation between adjacent image frames. Sincethe transducer is rotated about its long axis between adjacent imageframes, the spatial separation between points and lines in the twoimages depends upon their distance from the transducer and the anglebetween the adjacent image frames, a factor that the processor easilycalculates using simple trigonometric formulas. The processor thengenerates a line between isolated edge points in adjacent image frames,which may appear when a structure (e.g., a valve surface) is imaged edgeon in the two-dimensional image frames. The processor generates asurface between lines in adjacent image frames, which will occur when astructure is imaged face on (e.g., the septum 16 imaged from the rightventricle as illustrated in FIG. 2). This process can be repeated forall adjacent image frames until a full three-dimensional set of linesand surfaces have been defined that link together all of the averagespatial locations of detected edges in the dataset. The result is a lineand surface rendition of the tissue structures imaged in all of theultrasound images.

The processor may use a variety of algorithms to connect adjacentcartoon rendered image frames. The processor can recognize that pointsor lines in one image frame are part of the same structure imaged in anadjacent frame by comparing their separation distance to thresholdvalues. Then, the processor can determine a line or surface whichconnects related points and lines together to span the volume containedwithin and between the adjacent image frames. The connecting line orsurface can be stored as a vector or tensor value in order to reduce theamount of data required to describe the structure in three-dimensionalspace.

Instead of working from two-dimensional cartoon renderings, theprocessor in step 1009 can work directly from the three-dimensionalspatial location dataset to generate a three-dimensional cartoonrendering of the imaged volume. Information regarding the transducerrotational orientation stored or correlated with the image frames isused to determine the spatial separation between adjacent image frames.A variety of algorithms may be used to connect average edge locations inadjacent image frames to generate the cartoon lines and surfaces in step1009. In a first example algorithm, the processor first determines thecorrespondence of the points and lines in adjacent image frames, such asby comparing the spatial coordinates of the points/lines to determinewhether two points/lines are positioned closely together (i.e., within atolerance limit) in the two frames, and then calculating a vector orsurface which links together the corresponding points and lines throughthe space between adjacent image frames. Algorithms for comparingspatial locations within a tolerance and interpolating between twospecial coordinates are well known in the computer programming arts orwithin the ordinary skill of a computer programmer. A second examplealgorithm compares the spatial location information in two or more, oreven all image frames and determines best-fit lines and surfaces thatconnect the spatial locations together. Algorithms for generatingbest-fit lines and surfaces for connecting spatial location data arewell known in the computer programming arts. This process may beiterative by which the processor makes a first estimate of the best-fitfrom frame to frame, and then repeats the process across all frames tosmooth out discontinuities and inconsistent interpolations. A thirdexample algorithm assembles all image frames into a three-dimensionalrendition and then generates intra-frame connecting lines and surfacesby geometric interpolation.

As part of generating a three-dimensional cartoon rendering, step 1009,the processor may smooth out the connecting lines and surfaces among theimage frames. For example, the processor may test the rendered lines andsurfaces for sudden changes in direction. If a line or surface makes adeviation that exceeds a threshold (e.g., exhibiting an angle greaterthan a threshold), the processor may adjust the shape of the line orsurface to provide a smooth transition, such as by implementing aquadratic or cubic best-fit curve between or among points in three orfour adjacent image frames. Alternatively, the processor may use aspatial location value that is slightly removed from the average spatiallocation in a particular frame in order to enable a smooth connectionacross three or four adjacent image frames. In determining whether linesand surfaces should be smoothed, the processor may employ evaluationcriteria that take into account knowledge of the properties and nominalshapes of heart tissues and structures.

The result of the three-dimensional cartoon rendering step 1009 may be anew three-dimensional spatial dataset of lines and surfaces. Thisdataset may be much smaller in terms of stored information as lines andsurfaces may be represented as vector and tensor values, therebyeliminating the need to store individual spatial location data from allof the image frames.

With a three-dimensional cartoon rendering dataset in memory, theprocessor can then generate a three-dimensional display of thestructures, step 1008. Any of a number of well known methods fordisplaying three-dimensional image data may be employed, includingperspective projections, rotating displays, and adjacent rotatedperspective views. Also, well known display methods and algorithms maybe used to allow the clinician to manipulate the display, such asrotating the image, zooming in or away from the image, and moving intoor out of the image.

To use this embodiment method, a clinician may obtain several ultrasoundimages of the heart over a few cardiac cycles and then rotate theultrasound transducer (e.g., by rotating the handle of the catheterslightly) to change the imaging perspective, particularly the rotationalorientation. The ultrasound images are stored in a dataset that includesor is correlated to data indicating the rotational orientation of thetransducer. By imaging over a few cardiac cycles, rotating the catheter,and then repeating the process, the clinician can obtain a large datasetof two-dimensional ultrasound slice images spanning a partial orcomplete rotation of the transducer array that can be processed usingthis embodiment method. Optionally, ECG data may also be recorded in thedataset for selecting image frames which correspond to like portions ofthe ECG trace for further processing together.

While the foregoing method assumes that the processing is accomplishedafter the ultrasound images have been obtained, i.e., in post-procedureprocessing, the method can also be used to obtain and generate livecartoon rendered images, as illustrated in FIG. 11. In this alternativeembodiment, the steps of receiving ultrasound images from the ultrasoundimaging system 1002, detecting structure edges 1003, and determining thespatial location of detected edges 1005, are performed as describedabove with reference to FIG. 10. Instead of averaging the spatiallocations of detect edges in all images, a moving average of the spatialpositions of detected edges is obtained from a subset of the stream ofimages, step 1101. In this step, the spatial location data from aselected number (greater than two) of ultrasound image frames in aseries of image frames are used to determine nominal special locationdata using statistical methods (e.g., average, mean or other statisticalmeasure as described above) in a manner that updates as more images areobtained. For example, the nominal position data may be obtained bytaking the average of the spatial position data in five consecutiveimage frames, with the average being updated with each subsequent frameby including its spatial data in the average and disregarding data fromframes older than five frames back in time. The result is a movingaverage of spatial location data that is used as the nominal position ofthe detected edge. The number of image frames used in calculating themoving average may be anything more than two frames, although it isbelieved that averaging more that five to seven frames worth of spatialdata is unlikely to result in a significantly more representative oraccurate representation of the heart.

Using the moving spatial average positional data derived in step 1101,the processor can identify and reject outlier images or spatial positiondata, step 1006. The processor can also generate a cartoon rendering ofthe spatial average edge position data, step 1007, that is provided as adisplay for the clinician, step 1008, using methods similar to thosedescribed above with reference to FIG. 10.

This alternative embodiment is well suited for generating a real-timenominal image of a rapidly moving heart, such as a diseased heart infibrillation. In this embodiment, the cartoon rendering of image framesis accomplished at the same time as images are obtain. Consequently,there will not be a three-dimensional spatial dataset available forsimultaneously generating a three-dimensional composite image.Accordingly, the processor may store the cartoon rendered spatialinformation (e.g., vector and tensor data) in a dataset as such data isgenerated, step 1102. The stored cartoon rendered spatial informationmay include or be correlated with data describing the rotationalorientation and position of the transducer array. With such a dataset,the processor can then combine the cartoon renderings to create athree-dimensional image or dataset, step 1009, according to the variousmethods described above with reference to FIG. 10.

Methods for Processing Images to Remove Noise and Distortions:

Ultrasound image frames processed using the foregoing embodiment methodswill remove much of the noise and image distortions inherent inultrasound images. By recognizing structure and representing the beststructure location with a cartoon image, random pixels from noise,speckle, and multipath interference, as well as volumetric and timingdistortions can be significantly reduced or completely eliminated.Additionally, ultrasound images may be processed according to thefollowing embodiment methods in order to further overcome the noise anddistortion challenges inherent in ultrasound images.

As explained in U.S. Patent Publication No. 2005/0080336 A1 which waspreviously incorporated by reference, a number of images of the samestructure of the heart in the same shape (e.g., at rest or during thesame point within the cardiac cycle) can be used in a processing methodto average out, subtract out, or identify and ignore noise. Thesemultiple images can be averaged or combined by the imaging systemworkstation to provide a composite image having a greater signal tonoise ratio than present in any single image.

By way of example, each pixel in a plurality of images may be trackedand compared to a threshold occurrence level (e.g., number of images inwhich the pixel exceeds some threshold value) to eliminate spots (i.e.,bright pixels) that do not appear in at least a specified number ofimages. Each point or pixel in the various images can be inspected todetermine if an image value, or narrow range of values, is present inthe pixel in more than one image. If a particular pixel value (e.g., “0”indicating no echo) is present in a majority of images, a correspondingvalue can be assigned to the pixel in the compound image. In thismanner, a composite image can be assembled reflecting the most commonpixel values present in the various images. The threshold percentage ofimages containing a value required to set a corresponding pixel in thecomposite image may be adjusted from, for example, a significantminority, to a simple majority to a supermajority as desired to reducenoise or increase sensitivity. Alternatively, the value assigned to apixel may be based upon an average of the values for the pixel in thevarious images, with average values below an adjustable threshold set tozero to eliminate random speckle.

Noise pixels, such as speckle, will tend to be eliminated from acomposite image generated using this method because they occur randomlyin ultrasound images and therefore will not appear at a given pixel in amajority of images. Conversely, structure will tend to be enhanced insuch a composite image because echoes from structure will be received ina majority of images.

In an embodiment of the present invention, pixel values in a compositeimage may be established based upon pixel weighting factors applied tothe pixels in the processed images to obtain a weighted averageprocessed or composite image. Such weighting factors may be determinedstatistically based upon analysis of some or all of the ultrasoundimage, such as in an iterative or moving average analysis manner. Forexample, pixel values indicating the strength of the received echo(i.e., amplitude) may be used to generate a weighting factor foraveraging, such as large (i.e., bright) pixel values may be given ahigher weighting than pixels with low or dim pixel values. This approachrecognizes that structure is more likely to return a bright echo than arandom noise event. As another example, the weighting factor applied topixel averaging may vary as a function of distance from the transducerto compensate for the decline in signal-to-noise ratio with imagingdistance.

Methods for Processing Images to Register to an External ReferenceFrame:

Ultrasound image information can be processed to locate or register theimage data within an external frame of reference according to thefollowing method embodiment. An example of this embodiment isillustrated in FIG. 12. In this embodiment, the steps of receivingultrasound images from memory 1001 or the ultrasound imaging system1002, detecting structure edges 1003, determining the spatial locationof detected edges 1005, and generating a cartoon rendering of thespatial averaged structure are performed as described above withreference to FIG. 10. Using the cartoon rendered spatial structure data,the processor analyzes the structure to recognize particular features,step 1201. Be measuring the thickness and relative locations ofstructure, the processor can match the cartoon rendering to expectedstructure shapes and sizes as stored in a table of values or athree-dimensional digital model of the heart. In particular, the septumhas a thickness and is positioned prominently between the ventricles inmany imaging perspectives, so the septum can be easily identified by theprocessor based upon its wall thickness and/or position within the imageframe. For example, when imaging the left ventricle from within theright ventricle, the structure closest to the transducer will normallybe the septum. In other viewing perspectives, such as when viewingacross the atrial portion of the heart, other structures may berecognizable, such mitral valve or bicuspid valve. The processor notesthe spatial coordinates and orientation of the recognized structure(e.g., the septum).

Recognizing the septum or other structure in step 1201 providesinformation on the location of the ultrasound image within the heart,but not enough to accurately locate the image within the heart of thepatient. For example, if the processor determines that the septum islocated 2 cm from the transducer array, this does not locate thetransducer with respect to the long axis of the septum. To accuratelylocate the image, further structure must be recognize in step 1202. Inthis step, the processor evaluates the shape, position and orientationof other structure in the image with respect to the recognized structure(e.g., the septum) by comparing it to a three-dimensional digital modelof the heart. For example, in FIG. 2, the span of the ultrasound image26 encompasses a portion of the left ventricle wall 15 and a portion ofthe left atrium as well as the septum. Using the position andorientation of the septum with respect to other structure, the processorcompares their sizes and relative positions and orientations to athree-dimensional digital model of the heart to determine a most likelylocation of the image. This pattern matching may be accomplished bygeometric transformation algorithms which manipulate the digital modelof the heart until a close match with the cartoon rendered structure isdetermined. Based upon this match, the processor can locate theultrasound image frame within the heart.

The processor can then align, locate or otherwise register theultrasound image within the patient, step 1203. This may be accomplishedby recognizing structures, such as a ventricle wall, that has a knownlocation within the patient. Alternatively, a recognized structure, suchas the septum, may also be located using fluoroscopy, so the position ofthe cartoon rendering can be correlated to the external frame ofreference of the fluoroscopy system. In yet another alternative, thetransducer itself may be imaged by fluoroscopy, allowing its position tobe determined within the external frame of reference of the fluoroscopysystem. Also, other devices located in, on or near the heart, such aselectrophysiology catheters or pacemaker leads may appear in both theultrasound image and a fluoroscope image, allowing the cartoon renderingto be registered directly with the fluoroscopy system. Additionally, twoor more of these methods may be employed simultaneously to moreaccurately align the ultrasound image within the external referenceframe.

In an alternative embodiment illustrated in FIG. 13, the process ofrecognizing structures and correlating them to a three-dimensional modelof the heart can be used to also determine the imaging perspective ofthe transducer. In this embodiment, the steps of receiving ultrasoundimages from memory 1001 or the ultrasound imaging system 1002, detectingstructure edges 1003, determining the spatial location of detected edges1005, and generating a cartoon rendering of the spatial averagedstructure are performed as described above with reference to FIG. 10.Using the cartoon rendered spatial structure data, the processor thenmatches the cartoon rendering to a three-dimensional digital model ofthe heart, step 1301. This matching of the cartoon rendering to themodel heart may be accomplished using the methods described above withreference to FIG. 12. Alternatively, all of the structure lines in thecartoon rendering may be compared to the three-dimensional digital heartmodel, such as by rotating and slicing the model using a variety ofgeometric transformations, until a best fit position and orientation isidentified.

Once a best fit of the cartoon rendering within the three-dimensionaldigital heart model is obtained, the processor can register the cartoonrendering within the patient or within an external frame of reference,step 1203, using some or all of the methods described above withreference to FIG. 12.

Using the position of the cartoon rendering within the three-dimensionalheart model, the processor can determine the transducer location andorientation by back-calculating, step 1302. In this step, the distanceto and orientation of recognized structure, along with the actualposition of this structure in the heart model, are used to estimate thepoint of origin of the image. Any of a number of image processingalgorithms may be used for this process, including ray tracing andtriangulation techniques.

The processor can then store the calculated imaging perspective (i.e.,transducer position in space and orientation with respect to its threeaxes of rotation) along with the cartoon rendered image in an imagedataset, step 1303. This information may be stored within the dataset,such as the position and orientation of the point of origin of thecartoon image, or stored as a correlated dataset linked to the imagedataset by a shared key, pointer or other reference.

Finally, after all of the ultrasound images have been received andprocessed, the processor can use the information in the resultingdataset to generate a three-dimensional cartoon rendering of the heart,step 1009, using any of the methods described above with reference toFIGS. 10 and 11.

In a variation to this embodiment, the precise transducer array positionand orientation information obtained by various embodiments may becombined with the structure registration information obtained by imageprocessing in order to more accurately align or register thetwo-dimensional, three-dimensional or four-dimensional ultrasound imageswithin the patient or with respect to an external frame of reference. Inthis embodiment, the estimated transducer array position and orientationinformation provided by sensors built into the catheter may be combinedwith X-ray or computer tomography (CT) scan data to more accuratelyregister the ultrasound image data within the X, Y, Z coordinates of apatient-centered or external centered frame of reference. In thismanner, structures detected in the ultrasound images (i.e., sources ofultrasound echoes) can be located at relatively precise points (e.g., atspecific X, Y, Z coordinates) within the external frame of reference. Inthis manner, registration errors can be reduced by essentially combiningtwo or more independent methods for correlating or registering theultrasound images with respect to an external coordinate frame ofreference. By correlating or registering the two-dimensional,three-dimensional or four-dimensional ultrasound image sets within apatient or external frame of reference, the ultrasound image data maythen be fused with other image data (such as X-ray or CT scan data) toproduce high quality, multi-sensor images of the patient.

One method for locating ultrasound transducers within an external frameof reference employs fluoroscopy to image the catheter while ultrasoundimages are obtained. Methods for locating the position and orientationof ultrasound transducer arrays within a patient using fluoroscopy aredisclosed in U.S. patent application Ser. No. 11/610,386 previouslyincorporated by reference. In such methods, the X-ray source and imagingplane are at known coordinates within an external frame of reference, sothe transducer array position and orientation can be determined withrespect to that frame of reference. Using this information, theprocessor can correlate the ultrasound images taken at the same time thetransducer array is localized using fluoroscopy to the external frame ofreference by coordinate transformations using well known algorithms.

Solutions for Processing Images to Provide an Improved DisplayInterface:

Using image data obtained according to the various aforementionedembodiments an improved display of ultrasound images can be provided toenable clinicians to study details of the display from differentperspectives. An example of this embodiment is illustrated in FIG. 14.Having generated a three-dimensional dataset of heart structures withinthe imaged volume, a processor can manipulate this data to generatedisplays of the heart according to a clinician's specifications.

Referring to FIG. 14, the process steps of receiving ultrasound imagesfrom memory 1001 or the ultrasound imaging system 1002, detectingstructure edges 1003, determining the spatial location of detected edges1005, and generating a cartoon rendering of the spatial averagedstructure are performed as described above with reference to FIG. 10.Similarly, the process steps of recognizing a reference structure likethe septum 1201, pattern matching the recognized structure to otherstructures and to a three-dimensional digital model of the heart, step1202, and registering the cartoon rendered image to the patient or anexternal frame of reference, step 1203, are performed as described abovewith reference to FIG. 12. The cartoon rendered images can then beassembled into a three-dimensional dataset, step 1009, using the methodsdescribed above with reference to FIG. 10. As noted above, the result ofthese image processing steps will be a three-dimensional dataset thatsummarizes the structural information that could be derived from a largenumber of ultrasound image frames. To provide greater utility to theclinician, the processor can be configured by software programming toprovide an interactive display.

Providing the interactive display includes providing a display of thethree-dimensional cartoon rendered image while providing the user withan interface for designating, zooming or selecting a portion of thethree-dimensional image for viewing, step 1401, and then generating adisplay showing the selected portion of the three-dimensional image,1402. The user interface may be any sort of user interface well known inthe computer science arts, including a touch screen, graphical userinterface (GUI) with pointer (e.g., a mouse or light pen), and/or a setof user commands enterable via a keyboard input device. The userinterface may include menus on the display for indicating the optionsthe clinician has for investigating the three-dimensional image dataset.The display included with the user interface may be a view of thecartoon rendered three-dimensional image, which may be rotated, zoomedin or out, or cut into cross sections in response to user inputs, suchas by a pointer or keyboard input device.

In providing the display interface, the processor may also showinformation relating the displayed image to an external reference frame,such as with respect to the patient's body. In doing so, the processormay provide the clinician with options for viewing portions of thecartoon rendering from perspectives related to the patient's body orexternal equipment. For example, the clinician may use the informationin the display interface to view a portion of the heart as it would beseen during open heart surgery (i.e., from the chest) or the perspectiveimaged by a fluoroscopy system.

In response to inputs from the clinician, the processor accesses thecartoon rendered three-dimensional image dataset and calculates aperspective that corresponds to the requested view. Using the calculatedperspective, the processor then generates a view of the cartoonrendering as would be viewed from that perspective sized to fill thearea of the display. Since the cartoon rendering is stored as a vectoror tensor dataset, the generation of the perspective view can beaccomplished using geometric transformation algorithms well know in thecomputer science arts.

This embodiment allows the clinician to fully interrogate theinformation contained within the ultrasound image data including theability to view structure from perspectives not directly viewed inultrasound images and with zoom capability without the distraction ofultrasound noise and speckle.

Overall Image Processing Procedures.

The foregoing imaging and image processing embodiments may be usedtogether or in sequence to provide diagnostically useful information. Ina typical ultrasound imaging procedure, the clinician will position theultrasound imaging catheter within the heart of the patient usingfluoroscopy (or other means) to guide the transducer to the desiredlocation. The transducer location and orientation may be confirmed bythe clinician viewing some preliminary ultrasound images on the systemdisplay. When the clinician is satisfied with the ultrasoundtransducer's viewing perspective, the clinician may initiate the captureof ultrasound images, recording of the transducer location androtational orientation data, and (optionally) recording ECG data (e.g.,using electrophysiology equipment). The ultrasound image, transducerlocation/orientation data and ECG data are stored in processor memory orconnected data recording equipment. When sufficient ultrasound imageshave been obtained from a particular viewing perspective, such asspanning several heartbeats, the clinician may rotate the transducerthrough a small angle to give it a new viewing perspective. Then theprocess of recording ultrasound images, transducer location/orientationdata and ECG data is repeated until a sufficient number of images havebeen obtained. This process of rotating and imaging can be repeateduntil the catheter has been rotated through the entire angle of rotationrequired to image the desired volume of the patient's heart.

While gathering ultrasound images, the processor may generate anddisplay a cartoon rendering of the ultrasound images at each viewingperspective in order to provide the clinician with a cartoon view of thestructure being imaged. The clinician may then switch between viewingraw ultrasound data and viewing the cartoon rendered structure.Additionally, the clinician may direct the processor to correlate thecartoon rendered image to an external reference frame, such as thepatient or examination table.

Once all of the ultrasound image data for the desired imaging volume hasbeen obtained, the clinician may direct the processor to generate athree dimensional cartoon rendered image dataset. This dataset may bedisplayed for viewing by the clinician who may then use an interactiveinterface to select a particular portion for closer viewing. By viewingthe selected display, the clinician may decide that additional imagesshould be obtained, such as to improve the image clarity of a particularportion, or that the transducer should be repositioned in order toobtain images from a different perspective, such as to investigate aparticular volume of interest not well imaged. The clinician may followa similar procedure in obtaining the additional ultrasound images.

Once the ultrasound imaging procedure is over, the clinician may use theprocessor to interrogate the three-dimensional image dataset in order toassess the health of the patient's heart. For example, the clinician mayuse the interactive display in order to view various portions of theheart from different viewing angles and at different magnifications. Theclinician may also use the processor to generate a four-dimensional view(i.e., moving three-dimensional image) by selecting ultrasound imagesfor processing based upon the ECG signal at the time of each image. Thevarious processing embodiments may be employed on images for processingbased on ECG triggering in order to generate three-dimensional cartoonrenderings at particular portions of the heartbeat cycle.

In each of the foregoing embodiments, image frames may be selected forprocessing based upon trigger events or like portions of ECG data sothat cartoon rendered image frames correspond to like portions of theheartbeat cycle as well as approximately equal transducer rotationalorientations. In such processing, the process steps illustrated in FIG.10-14 may be repeated for each of different trigger events, or varyingtime delays after a recognizable trigger event (e.g., the R wave). Inthis manner, a three-dimensional cartoon rendered image can be generatedfor the heart at the time of a particular repeating trigger event orlike portion of the ECG trace. Also, a four-dimensional (three spatialdimensions plus time) cartoon rendered image dataset can be generatedfor the heart spanning a portion or all of the heartbeat cycle.

In another embodiment, three-dimensional and four-dimensional imageultrasound image datasets generated according to various embodiments maybe combined with image data from one or more external sources. Forexample, fluoroscopic images of the heart may be correlated toultrasound images using time stamp or ECG data and thereby correlated toparticular three-dimensional cartoon rendered ultrasound image datasets.Such correlated images then can be presented as overlapping or otherwisemerged images on a display. Since the positions of the X-ray source andimaging plane are known within an external frame of reference, thiscomposite display will show the cartoon rendered structures overlappingor otherwise fixed in the external frame of reference. This embodimentmethod may also enable clinicians to see structures outside of theultrasound image scan (e.g., behind the transducer array or beyond theimaging range of the transducer) as they match up with ultrasound imagedstructures. In this manner, the physician may locate the ultrasoundimages or the cartoon rendered images of heart structure with respect toribs, vertebrae, implanted pacemakers and pacing leads, or othercatheters (such as ablation catheters) that are imaged by fluoroscopy.

In another embodiment, externally applied ultrasound localizingequipment with fiducial references may be employed to locate theintracardiac catheter in two correlated coordinate systems. In thisembodiment, three or more ultrasound transducers may be positioned onthe patient at locations correlated to the external frame of reference.By imaging the catheter using the three or more external ultrasoundtransducers, the location of the catheter in three-dimensional spacewith respect to the external transducers can be determined by echolocation. The processor can then locate the intracardiac ultrasoundimages within the external frame of reference by means of two coordinatetransformations (image to external transducer frame of reference, andthen external transducer frame of reference to the external frame ofreference). An example of a system of localizing ultrasound transducerssuitable for this method is provided in U.S. Pat. No. 5,515,853 which ishereby incorporated by reference in its entirety.

Each of the foregoing embodiment methods may be implemented on anultrasound imaging system embodiment, including system elementsdescribed above with reference to FIG. 1, with the system processoradapted and configured by software instructions to perform the variousmethod steps. Such implementing software instructions may be stored oncomputer readable memory accessed by the system processor, such assystem read only memory within the processor, system random accessmemory within the processor, internal hard drive memory, external harddrive memory, including an external hard drive memory coupled to thesystem processor via a network, and/or a compact disc.

Another embodiment of the present invention is a computer readablememory having software instructions stored therein which will direct aprocessor to perform the method steps of the foregoing embodimentmethods. Such a computer readable memory may be any storage mediumconnectable to a computer including, for example, read only memory,random access memory, internal hard drive memory, external hard drivememory, including an external hard drive memory coupled to andaccessible via a network, and a compact disc.

While the present invention has been disclosed with reference to certainexemplary embodiments, numerous modifications, alterations, and changesto the described embodiments are possible without departing from thesphere and scope of the present invention, as defined in the appendedclaims. Accordingly, it is intended that the present invention not belimited to the described embodiments, but that it have the full scopedefined by the language of the following claims, and equivalentsthereof.

I claim:
 1. A method for processing ultrasound images of tissue, comprising: receiving and storing a plurality of ultrasound image frames; receiving and storing a rotational orientation of a transducer corresponding to each of the plurality of ultrasound image frames; detecting edges of tissue structures within each of the plurality of image frames; determining a spatial location, relative to the transducer, of each of the detected edges of tissue structures in each of the plurality of image frames; receiving and storing an electrocardiogram signal corresponding in time of recording to the plurality of ultrasound image frames; performing the operations of statistically analyzing the spatial locations of the detected edges of tissue structures among a subset of the plurality of image frames with an approximately equal transducer rotational orientation and generating a cartoon rendered image frame of the tissue structures based upon representative spatial locations, relative to the transducer, imaged from the approximately equal transducer rotational orientation for a subset of the plurality of image frames corresponding to a trigger event in the electrocardiogram signal; and combining the plurality of cartoon rendered image frames within the subset of the plurality of image frames corresponding to the trigger event in the electrocardiogram signal using their respective corresponding approximately equal transducer rotation orientations to generate a three-dimensional cartoon rendering of tissue structure corresponding to the trigger event.
 2. The method of claim 1, wherein the statistically analyzing the spatial locations of the detected edges of tissue structures among the plurality of image frames comprises determining a weighted average location of the detected edges of tissue structures among the plurality of image frames.
 3. The method of claim 1, wherein the statistically analyzing the spatial locations of the detected edges of tissue structures among the plurality of image frames comprises calculating a moving average of the spatial locations of the detected edges among two or more image frames selected from the plurality of image frames.
 4. A computer for use in intracardiac ultrasound imaging, comprising: a processor configured with processor-executable instructions configured to cause the processor to perform operations comprising: storing a plurality of ultrasound image frames in memory; storing in memory a rotational orientation of a transducer corresponding to each of the plurality of ultrasound image frames; detecting edges of tissue structures within each of the plurality of image frames; determining a spatial location, relative to the transducer, of each of the detected edges in each of the plurality of image frames; receiving and storing an electrocardiogram signal corresponding in time of recording to the plurality of ultrasound image frames; performing the operations of statistically analyzing the spatial locations of the detected edges among a subset of the plurality of image frames with an approximately equal transducer rotational orientation and generating a cartoon rendered image frame of the tissue structures based upon representative spatial locations, relative to the transducer, imaged from the approximately equal transducer rotational orientation for a subset of the plurality of image frames corresponding to a trigger event in the electrocardiogram signal; and combining the plurality of cartoon rendered image frames within the subset of the plurality of image frames corresponding to the trigger event in the electrocardiogram signal using their respective corresponding approximately equal transducer rotation orientations to generate a three-dimensional cartoon rendering of tissue structure corresponding to the trigger event.
 5. The computer of claim 4, wherein the processor is configured with processor-executable instructions configured to cause the processor to perform operations further comprising time-gating the plurality of ultrasound image frames with the trigger event.
 6. The computer of claim 5, wherein the processor is configured with processor-executable instructions configured to cause the processor to perform operations further comprising time-gating the plurality of ultrasound image frames with the trigger event by using the trigger event to trigger at least one of the following: (a) generation of the plurality of ultrasound image frames, or (b) retrieval of stored ultrasound image frames. 